Ensure CMS works for Run3 and HL-LHC, through GMDL and CMSSW

[sethrj]

Ensure CMS works for Run3 and HL-LHC, through GMDL and CMSSW

[whokion]

List of all CMS CaloSD volumes from LH-LHC: cms-hllhc-CaloSD-volumes.txt

[sethrj]

@whokion I've added an updated list of the reachable volumes with attached sensitive detectors

To recap: our best guess is that CMSSW is attaching SDs to volumes that are not part of the world volume hierarchy: this only seems to be happening to the xtru detectors HGCalEECellSensitive0Fine and others: am0 and bm0 don't exist in the geometry, but dm0 and gm0 do. Both are written in the list of "volumes with SD attached".

[whokion]

With manually adding HL-LHC sensitive detectors:

#6  0x00007f1339b7f7f1 in G4Exception(char const*, char const*, G4ExceptionSeverity, char const*) () from /work1/g4gpu/syjun/cms/CMSSW_13_0_6/external/slc7_amd64_gcc11/lib/libG4global.so
#7  0x00007f133a3c971d in SensitiveDetector::cmsTrackInformation(G4Track const*) () from /work1/g4gpu/syjun/cms/CMSSW_13_0_6/lib/slc7_amd64_gcc11/libSimG4CoreSensitiveDetector.so
#8  0x00007f132a762079 in CaloSD::getTrackID(G4Track const*) () from /work1/g4gpu/syjun/cms/CMSSW_13_0_6/lib/slc7_amd64_gcc11/libSimG4CMSCalo.so
#9  0x00007f132a76a499 in CaloSD::ProcessHits(G4Step*, G4TouchableHistory*) () from /work1/g4gpu/syjun/cms/CMSSW_13_0_6/lib/slc7_amd64_gcc11/libSimG4CMSCalo.so
#10 0x00007f13363bd4ca in G4VSensitiveDetector::Hit (aStep=0x7f10a4ec0240, this=0x7f132a2ea500) at /work1/g4gpu/syjun/cms/externals/src/geant4.11.1.2/source/digits_hits/detector/include/G4VSensitiveDetector.hh:127
#11 celeritas::detail::HitProcessor::operator() (this=<optimized out>, out=...) at /work1/g4gpu/syjun/cms/externals/src/celeritas/src/accel/detail/HitProcessor.cc:242
#12 0x00007f1336e575a6 in celeritas::detail::StepGatherAction<(celeritas::StepPoint)1>::execute (this=0x7f12c1f5a300, params=..., state=...) at /work1/g4gpu/syjun/cms/externals/src/celeritas/src/celeritas/user/detail/StepGatherAction.cu:46

[whokion]

The log file with all calorimeter SDs added is test_find_volumes_many.log.

[whokion]

JSON diagnostic output file with a calorimeter-sd-only run:
celeritas-output.json.txt

[sethrj]

OK, even though the "find volume" is mapping e.g. HGCalEECellSensitive0Coares1a0 to a geant4 logical volume, both vecgeom and the GDML file are showing a volume named HGCalEECellSensitive0Coares2d1 (no 1a0 anywhere). I confirmed that the master and worker threads have the same name in the GDML file, so it's not random.

Could something be changing the names between the time that we call FindVolumes and the time VecGeom is initialized?

Even weirder is that these names seem to disagree only for the xtru shape volumes.

[whokion]

Back trace for the segmentation fault with all CMS HLHLC tracker sensitive volumes:

#0  0x00007fffb04fabec in TkAccumulatingSensitiveDetector::createHit(G4Step const*)
    ()
   from /cvmfs/cms.cern.ch/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_0_6/lib/slc7_amd64_gcc11/libSimG4CMSTracker.so
#1  0x00007fffb04fb726 in TkAccumulatingSensitiveDetector::ProcessHits(G4Step*, G4TouchableHistory*) ()
   from /cvmfs/cms.cern.ch/slc7_amd64_gcc11/cms/cmssw/CMSSW_13_0_6/lib/slc7_amd64_gcc11/libSimG4CMSTracker.so
#2  0x00007fffbc2cd84a in G4VSensitiveDetector::Hit (aStep=0x7ffd2ba8e2a0, 
    this=0x7fffb06af880)
    at /work1/g4gpu/syjun/cms/externals/src/geant4.11.1.2/source/digits_hits/detector/include/G4VSensitiveDetector.hh:127
#3  celeritas::detail::HitProcessor::operator() (this=<optimized out>, out=...)
    at /work1/g4gpu/syjun/cms/externals/src/celeritas/src/accel/detail/HitProcessor.cc:242
#4  0x00007fffbcd67786 in celeritas::detail::StepGatherAction<(celeritas::StepPoint)1>::execute (this=0x7fff45228260, params=..., state=...)
    at /work1/g4gpu/syjun/cms/externals/src/celeritas/src/celeritas/user/detail/StepGatherAction.cu:46
#5  0x00007fffbcc537b3 in celeritas::detail::ActionSequence::execute<(celeritas::MemSpace)1> (this=<optimized out>, params=..., state=...)
    at /work1/g4gpu/syjun/cms/externals/src/celeritas/src/celeritas/global/detail/ActionSequence.cc:111
#6  0x00007fffbcc4f9d9 in celeritas::Stepper<(celeritas::MemSpace)1>::operator() (
    this=0x7fffb0738a10)
    at /work1/g4gpu/syjun/cms/externals/src/celeritas/src/celeritas/global/Stepper.cc:58
#7  0x00007fffbc2b8a66 in celeritas::LocalTransporter::Flush (this=0x7fffb063a140)
    at /work1/g4gpu/syjun/cms/externals/src/celeritas/src/corecel/cont/detail/SpanImpl.hh:117
#8  0x00007fffc03132ca in EventAction::EndOfEventAction(G4Event const*) ()
   from /work1/g4gpu/syjun/cms/CMSSW_13_0_6/lib/slc7_amd64_gcc11/libSimG4CoreApplication.so
#9  0x00007fffbff7832d in G4EventManager::DoProcessing(G4Event*) ()
--Type <RET> for more, q to quit, c to continue without paging--
   from /work1/g4gpu/syjun/cms/CMSSW_13_0_6/external/slc7_amd64_gcc11/lib/libG4event.so
#10 0x00007fffc032c4da in RunManagerMTWorker::produce(edm::Event const&, edm::EventSetup const&, RunManagerMT&) ()

[whokion]

@whokion I've added an updated list of the reachable volumes with attached sensitive detectors

To recap: our best guess is that CMSSW is attaching SDs to volumes that are not part of the world volume hierarchy: this only seems to be happening to the xtru detectors HGCalEECellSensitive0Fine and others: am0 and bm0 don't exist in the geometry, but dm0 and gm0 do. Both are written in the list of "volumes with SD attached".

The clean list cleared the exception, but hits another at HGCalSD due to missing information:

#0  0x00007fffb886fb26 in HGCalSD::setDetUnitId(G4Step const*) ()
   from /work1/g4gpu/syjun/cms/CMSSW_13_0_6/lib/slc7_amd64_gcc11/libSimG4CMSCalo.so
#1  0x00007fffb884321e in CaloSD::ProcessHits(G4Step*, G4TouchableHistory*) ()
   from /work1/g4gpu/syjun/cms/CMSSW_13_0_6/lib/slc7_amd64_gcc11/libSimG4CMSCalo.so
#2  0x00007fffbc28b84a in G4VSensitiveDetector::Hit (aStep=0x7ffd2ba8e2a0, 
    this=0x7fffb0608000)
    at /work1/g4gpu/syjun/cms/externals/src/geant4.11.1.2/source/digits_hits/detector/include/G4VSensitiveDetector.hh:127
#3  celeritas::detail::HitProcessor::operator() (this=<optimized out>, out=...)
    at /work1/g4gpu/syjun/cms/externals/src/celeritas/src/accel/detail/HitProcessor.cc:242

[sethrj]

Here's the list of G4VSD daughter classes in CMS that I can find, with the ones we don't need commented out:

Calo/interface/ECalSD.h:31:class ECalSD : public CaloSD {
Calo/interface/HCalSD.h:38:class HCalSD : public CaloSD, public Observer<const BeginOfJob*> {
Calo/interface/HGCSD.h:22:class HGCSD : public CaloSD, public Observer<const BeginOfJob *> {
Calo/interface/HGCScintSD.h:19:class HGCScintSD : public CaloSD, public Observer<const BeginOfJob *> {
Calo/interface/HGCalSD.h:21:class HGCalSD : public CaloSD, public Observer<const BeginOfJob *> {
Calo/interface/HFNoseSD.h:21:class HFNoseSD : public CaloSD, public Observer<const BeginOfJob *> {
EcalTestBeam/interface/EcalTBH4BeamSD.h:21:class EcalTBH4BeamSD : public CaloSD {
HGCalTestBeam/interface/AHCalSD.h:14:class AHCalSD : public CaloSD {
HGCalTestBeam/interface/HGCalTB16SD01.h:15:class HGCalTB16SD01 : public CaloSD {
HcalTestBeam/interface/HcalTB02SD.h:31:class HcalTB02SD : public CaloSD {
HcalTestBeam/interface/HcalTB06BeamSD.h:20:class HcalTB06BeamSD : public CaloSD {
# CherenkovAnalysis/interface/DreamSD.h:22:class DreamSD : public CaloSD {
# Forward/interface/CastorSD.h:33:class CastorSD : public CaloSD {
# Forward/interface/ZdcSD.h:15:class ZdcSD : public CaloSD {
# Forward/interface/TotemT2ScintSD.h:9:class TotemT2ScintSD : public CaloSD {
Calo/interface/CaloSD.h:40:class CaloSD : public SensitiveCaloDetector,
Calo/interface/CaloTrkProcessing.h:27:class CaloTrkProcessing : public SensitiveCaloDetector,
ShowerLibraryProducer/interface/FiberSD.h:29:class FiberSD : public SensitiveCaloDetector,
ShowerLibraryProducer/interface/HFChamberSD.h:21:class HFChamberSD : public SensitiveCaloDetector {
ShowerLibraryProducer/interface/HFWedgeSD.h:20:class HFWedgeSD : public SensitiveCaloDetector {
Tracker/interface/TkAccumulatingSensitiveDetector.h:31:class TkAccumulatingSensitiveDetector : public SensitiveTkDetector,
# FP420/interface/FP420SD.h:44:class FP420SD : public SensitiveTkDetector,
# PPS/interface/TotemRPSD.h:30:class TotemRPSD : public SensitiveTkDetector, public Observer<const BeginOfEvent*>, public Observer<const EndOfEvent*> {
# PPS/interface/PPSPixelSD.h:46:class PPSPixelSD : public SensitiveTkDetector,
# PPS/interface/PPSDiamondSD.h:29:class PPSDiamondSD : public SensitiveTkDetector,
# Muon/interface/MuonSensitiveDetector.h:39:class MuonSensitiveDetector : public SensitiveTkDetector, public Observer<const BeginOfEvent*> {
# Forward/interface/TimingSD.h:29:class TimingSD : public SensitiveTkDetector, public Observer<const BeginOfEvent*> {
# Forward/interface/TotemSD.h:46:class TotemSD : public SensitiveTkDetector, public Observer<const BeginOfEvent*> {
# Forward/interface/BscSD.h:15:class BscSD : public TimingSD {
# Forward/interface/BHMSD.h:16:class BHMSD : public TimingSD {
# Forward/interface/Bcm1fSD.h:13:class Bcm1fSD : public TimingSD {
# Forward/interface/PltSD.h:13:class PltSD : public TimingSD {
# Forward/interface/MtdSD.h:19:class MtdSD : public TimingSD {
# Muon/interface/MuonSlaveSD.h:26:class MuonSlaveSD : public TrackingSlaveSD {

And as far as I can tell here are the properties, but I haven't done anything to exclude the debug statements other than filter out code with << in the line:

PostStepPoint->GetGlobalTime()
PostStepPoint->GetGlobalTime()
PostStepPoint->GetPosition()
PostStepPoint->GetTouchable()
PreStepPoint->GetCharge()
PreStepPoint->GetKineticEnergy()
PreStepPoint->GetMaterial()
PreStepPoint->GetMaterial()->GetDensity()
PreStepPoint->GetMaterial()->GetRadlen()
PreStepPoint->GetPhysicalVolume()
PreStepPoint->GetPhysicalVolume()->GetLogicalVolume()
PreStepPoint->GetPhysicalVolume()->GetLogicalVolume()->GetName()
PreStepPoint->GetPosition()
PreStepPoint->GetPosition().perp()
PreStepPoint->GetPosition().x()
PreStepPoint->GetPosition().y()
PreStepPoint->GetPosition().z()
PreStepPoint->GetTouchable()
PreStepPoint->GetTouchable()->GetReplicaNumber(0)
Track->GetDefinition()->GetParticleName()
Track->GetDefinition()->GetPDGCharge()
Track->GetDefinition()->GetPDGEncoding()
Track->GetGlobalTime()
Track->GetKineticEnergy()
Track->GetMomentum()
Track->GetMomentumDirection().theta()
Track->GetNextVolume()
Track->GetPosition()
Track->GetPrimaryTrack()
Track->GetTrackID()
Track->GetWeight()

Extracting uses

$ grep -Hnr '\(StepPoint\|Track|Step\)()->Get' Calo Tracker  | sort
# Vim: g/<Get([^;, ]+).*
# \1 \2->Get\3

All uses

Calo/plugins/CaloSteppingAction.cc:335: PreStepPoint->GetPhysicalVolume()->GetLogicalVolume()
Calo/plugins/CaloSteppingAction.cc:346: PreStepPoint->GetTouchable()
Calo/plugins/CaloSteppingAction.cc:347: PreStepPoint->GetPosition()
Calo/plugins/CaloSteppingAction.cc:356: PreStepPoint->GetCharge()
Calo/plugins/CaloSteppingAction.cc:357: PreStepPoint->GetMaterial()->GetDensity()
Calo/plugins/CaloSteppingAction.cc:378: PreStepPoint->GetMaterial()->GetRadlen()
Calo/plugins/CaloSteppingAction.cc:383: PreStepPoint->GetCharge()
Calo/plugins/CaloSteppingAction.cc:384: PreStepPoint->GetMaterial()->GetDensity()
Calo/plugins/CaloSteppingAction.cc:395: PreStepPoint->GetTouchable()
Calo/plugins/CaloSteppingAction.cc:396: Track->GetGlobalTime()
Calo/plugins/CaloSteppingAction.cc:397: Track->GetTrackID()
Calo/plugins/CaloSteppingAction.cc:398: Track->GetDefinition()->GetPDGEncoding()
Calo/plugins/CaloSteppingAction.cc:400: PreStepPoint->GetPosition().x()
Calo/plugins/CaloSteppingAction.cc:401: PreStepPoint->GetPosition().y()
Calo/plugins/CaloSteppingAction.cc:402: PreStepPoint->GetPosition().z()
Calo/plugins/CaloSteppingAction.cc:411: Track->GetNextVolume()
Calo/plugins/CaloSteppingAction.cc:413: PreStepPoint->GetKineticEnergy()
Calo/plugins/CaloSteppingAction.cc:415: PostStepPoint->GetGlobalTime()
Calo/plugins/CaloSteppingAction.cc:435: Track->GetPosition()
Calo/plugins/CaloSteppingAction.cc:437: Track->GetMomentum()
Calo/plugins/CaloSteppingAction.cc:441: PreStepPoint->GetPhysicalVolume()
Calo/plugins/HcalTestAnalysis.cc:313: PreStepPoint->GetPhysicalVolume()
Calo/plugins/HcalTestAnalysis.cc:351: Track->GetDefinition()->GetParticleName()
Calo/plugins/HcalTestAnalysis.cc:353: PreStepPoint->GetPosition().x()
Calo/plugins/HcalTestAnalysis.cc:354: PreStepPoint->GetPosition().y()
Calo/plugins/HcalTestAnalysis.cc:355: PreStepPoint->GetPosition().z())
Calo/src/CaloSD.cc:172: PreStepPoint->GetTouchable()))
Calo/src/CaloSD.cc:183: PreStepPoint->GetPhysicalVolume()
Calo/src/CaloSD.cc:269: Track->GetPrimaryTrack()
Calo/src/CaloSD.cc:436: PreStepPoint->GetPosition()
Calo/src/CaloSD.cc:678: PreStepPoint->GetCharge()
Calo/src/CaloSD.cc:682: PreStepPoint->GetMaterial()->GetDensity()
Calo/src/CaloTrkProcessing.cc:185: PreStepPoint->GetTouchable()
Calo/src/CaloTrkProcessing.cc:186: PostStepPoint->GetTouchable()
Calo/src/ECalSD.cc:457: PreStepPoint->GetTouchable()
Calo/src/HCalSD.cc:337: PreStepPoint->GetTouchable()->GetReplicaNumber(0))
Calo/src/HCalSD.cc:350: PreStepPoint->GetPosition()
Calo/src/HCalSD.cc:374: PreStepPoint->GetPhysicalVolume()->GetLogicalVolume()->GetName()
Calo/src/HCalSD.cc:382: PreStepPoint->GetPhysicalVolume()->GetLogicalVolume()->GetName()
Calo/src/HCalSD.cc:401: PreStepPoint->GetPhysicalVolume()->GetLogicalVolume()
Calo/src/HCalSD.cc:453: PreStepPoint->GetTouchable()
Calo/src/HCalSD.cc:476: PreStepPoint->GetPosition()
Calo/src/HCalSD.cc:515: PreStepPoint->GetMaterial()
Calo/src/HCalSD.cc:634: PreStepPoint->GetTouchable()
Calo/src/HCalSD.cc:868: PostStepPoint->GetGlobalTime())
Calo/src/HCalSD.cc:927: PostStepPoint->GetGlobalTime())
Calo/src/HCalSD.cc:999: PreStepPoint->GetTouchable()
Calo/src/HFCherenkov.cc:65: Track->GetWeight()
Calo/src/HFCherenkov.cc:77: PreStepPoint->GetPosition())
Calo/src/HFCherenkov.cc:79: PostStepPoint->GetPosition())
Calo/src/HFGflash.cc:155: PreStepPoint->GetCharge()
Calo/src/HFNoseSD.cc:103: Track->GetWeight()
Calo/src/HFNoseSD.cc:139: PreStepPoint->GetMaterial()
Calo/src/HFNoseSD.cc:150: PreStepPoint->GetMaterial()->GetRadlen()
Calo/src/HFNoseSD.cc:84: PreStepPoint->GetPosition().perp()
Calo/src/HFNoseSD.cc:85: PreStepPoint->GetPosition().z())
Calo/src/HFNoseSD.cc:87: Track->GetDefinition()->GetPDGEncoding()
Calo/src/HFNoseSD.cc:88: Track->GetDefinition()->GetParticleName()
Calo/src/HFNoseSD.cc:89: PreStepPoint->GetPhysicalVolume()->GetLogicalVolume()
Calo/src/HFShower.cc:111: PostStepPoint->GetGlobalTime())
Calo/src/HFShower.cc:180: Track->GetDefinition()->GetPDGCharge()
Calo/src/HFShower.cc:220: PostStepPoint->GetGlobalTime())
Calo/src/HFShower.cc:269: Track->GetDefinition()->GetPDGCharge()
Calo/src/HFShower.cc:329: PostStepPoint->GetGlobalTime())
Calo/src/HFShower.cc:51: Track->GetDefinition()->GetPDGCharge()
Calo/src/HFShowerParam.cc:302: PostStepPoint->GetGlobalTime())
Calo/src/HGCSD.cc:102: Track->GetDefinition()->GetPDGEncoding()
Calo/src/HGCSD.cc:103: PreStepPoint->GetPhysicalVolume()->GetLogicalVolume()
Calo/src/HGCSD.cc:116: Track->GetWeight()
Calo/src/HGCSD.cc:122: PreStepPoint->GetTouchable()
Calo/src/HGCSD.cc:123: Track->GetKineticEnergy()
Calo/src/HGCSD.cc:124: Track->GetDefinition()->GetPDGEncoding()
Calo/src/HGCSD.cc:125: Track->GetMomentumDirection().theta())
Calo/src/HGCSD.cc:170: PreStepPoint->GetMaterial()
Calo/src/HGCSD.cc:183: PreStepPoint->GetMaterial()->GetRadlen()
Calo/src/HGCSD.cc:98: PreStepPoint->GetPosition().perp()
Calo/src/HGCSD.cc:99: PreStepPoint->GetPosition().z())
Calo/src/HGCScintSD.cc:119: PreStepPoint->GetPosition().perp()
Calo/src/HGCScintSD.cc:120: PreStepPoint->GetPosition().z())
Calo/src/HGCScintSD.cc:122: Track->GetDefinition()->GetPDGEncoding()
Calo/src/HGCScintSD.cc:123: Track->GetDefinition()->GetParticleName()
Calo/src/HGCScintSD.cc:124: PreStepPoint->GetPhysicalVolume()->GetLogicalVolume()
Calo/src/HGCScintSD.cc:138: Track->GetWeight()
Calo/src/HGCScintSD.cc:185: PreStepPoint->GetMaterial()
Calo/src/HGCScintSD.cc:196: PreStepPoint->GetMaterial()->GetRadlen()
Calo/src/HGCalSD.cc:101: PreStepPoint->GetPosition().perp()
Calo/src/HGCalSD.cc:102: PreStepPoint->GetPosition().z())
Calo/src/HGCalSD.cc:104: Track->GetDefinition()->GetPDGEncoding()
Calo/src/HGCalSD.cc:105: Track->GetDefinition()->GetParticleName()
Calo/src/HGCalSD.cc:106: PreStepPoint->GetPhysicalVolume()->GetLogicalVolume()
Calo/src/HGCalSD.cc:120: Track->GetWeight()
Calo/src/HGCalSD.cc:162: PreStepPoint->GetMaterial()
Calo/src/HGCalSD.cc:173: PreStepPoint->GetMaterial()->GetRadlen()
Tracker/src/TkAccumulatingSensitiveDetector.cc:123: PreStepPoint->GetTouchable())
Tracker/src/TkAccumulatingSensitiveDetector.cc:281: PostStepPoint->GetPosition())
Tracker/src/TkAccumulatingSensitiveDetector.cc:305: PostStepPoint->GetPosition())

Notes:

• PostStepPoint->GetTouchable is only used in doFineCalo_ logic
• Some of these accessors are derivative from other data (e.g., PhysicalVolume) while others are not (e.g., Charge)

[sethrj]

@whokion confirms that we can run CMSSW:

• both HL and run 3 configurations
• with an RZ magnetic field, and uniform
• with celeritas sending data back to Geant4 SDs for em+hadronic calo+tracker + hf

Yay! Now we just have to get it to run fast...