marinang/simprod

Mini framework to send LHCb simulation jobs to lxplus or a slurm batch system!

simprod's Issues

Monitoring not working properly

Hi,

I can't seem to get the monitoring to work. When I try the command jobs I get:

/afs/cern.ch/user/j/jdeabreu/Kmutau/SimulationProduction/simprod/simjob/simjob.pyc in __str__(self)
    138                 nevents = job_doc["nevents"]
    139                 subjobs = job_doc["nsubjobs"]
--> 140                 nrunning = job_doc["nrunning"]
    141                 ncompleted = job_doc["ncompleted"]
    142                 nfailed = job_doc["nfailed"]

KeyError: 'nrunning'

I can see individual jobs with say jobs[1], but using jobs alone doesn't work. Another example is jobs.select('running')

/afs/cern.ch/user/j/jdeabreu/Kmutau/SimulationProduction/simprod/simjob/simjob.pyc in select(self, status)
    222
    223     def select(self, status):
--> 224         return self.jobcollection.get(self.query.status == status)
    225
    226     def _update(self, in_init = False):

AttributeError: 'JobCollection' object has no attribute 'query'

Maybe I'm doing something stupid. I can at least submit jobs and they do finish.

Json file get corrupted when simprod is not closed properly

From Mattermost:
The jobs file simprod.simjob get corrupted when simprod is not closed properly.
The spurious characters at the end of the file can be removed by hand, but a better solution would be useful.

Possible solutions:

• the corrupted json files always has the same patterns, so a function can be designed to remove the additional characters.
• having a folder with the history copy of the simjobs.json at different times, such that if the last one gets corrupted the previous one can be read.

Err and Out deleted after completion?

Is it on purpose that the err and out files are removed for completed jobs? It's sometimes quite useful to have these as well.

Turbo production.

I am not a user of Turbo lines but I've implemented the Tesla steps in the MC production. So if someone could test the output and check that everything is alright that would be great!

How to use .lhe files as generator files

Hi,

I'm Davide, LHCb PhD student and I would like to use this tool to do some private production.
Is it possible to give as input some .lhe files generated with MadGraph?
And if so, how should I do it?

Thank you very much,
Davide

Can't init grid proxy anymore

Hello,

I just gave a shot to SimProd and submitted a test job from lxplus. I had a ganga session running from a different lxplus machine, which crashed. Since then, I can't initialise my proxy in any new session (see below). There must have been some interference with grid proxies or setting SimProd changed some default locations/variables. Any solution to fix this simply?

Best,

Max

[chefdevi@lxplus797 ~]$ lhcb-proxy-init

Traceback (most recent call last):
File "/cvmfs/lhcb.cern.ch/lhcbdirac/versions/v10.4.9-1647535249/Linux-x86_64/bin/lhcb-proxy-init", line 8, in
sys.exit(main())
File "/cvmfs/lhcb.cern.ch/lhcbdirac/versions/v10.4.9-1647535249/Linux-x86_64/lib/python3.9/site-packages/DIRAC/Core/Utilities/DIRACScript.py", line 65, in call
entrypoint = min(
File "/cvmfs/lhcb.cern.ch/lhcbdirac/versions/v10.4.9-1647535249/Linux-x86_64/lib/python3.9/site-packages/DIRAC/Core/Utilities/DIRACScript.py", line 67, in
key=lambda e: rankedExtensions.index(entrypointToExtension(e)),
ValueError: 'LbDiracWrappers' is not in list

Server responded with an error: [3011]

Everytime a file moves from running to completed, I get the following error message which does not seem to have any consequence though - the files are at the correct place and seem to be fine.
INFO Moving subjob 7.126 to final destination!
[ERROR] Server responded with an error: [3011] Unable to rename - source does not exist

INFO status of subjob 7.126 changed from 'running' to 'completed'

Resubmit method

A resubmit method as in ganga would be nice! Such that one can loop over the subjobs and resubmit the failed ones. Sorry for the noise if this already exists and I didn't find it.

Downgrade to tinydb v3.11.1

If you have at some point an error like this:

    return self._query_cache[cond][:]
TypeError: 'NoneType' object has no attribute '__getitem__'

It is expected, see msiemens/tinydb#248. It will be fixed but in the meantime please downgrade tinydb to v3.11.1

Matt

Old version of decfiles is loaded

The getevttype method gets a rather old version of the requested decfile. Should the default be the most recent version?