Implement initial electromagnetic physics

[celeritas-project]

Implement initial electromagnetic physics

[whokion]

add @mrguilima for Rayleigh.

[vrpascuzzi]

As per the discussion with @sethrj, I'll start working on the BetheHeitler (pair production) model.

[sethrj]

@stognini @whokion @pcanal Are you able to participate in the physics "virtual collocation" on Wednesday? If so, please select a process above by tomorrow so that I can get the file templates set up for all of us beforehand. Thanks!

[whokion]

• added Rayleigh and assigned it to whokion (Soon Yung Jun).
• Assigned eplus2gmma to whokion

[whokion]

@sethrj No problem. I will take care of those two process classes before diving into a non-uniform magnetic field application.

[sethrj]

I think this is actually "complete"! There are still a few little pieces (immediately thinking positron xs scaling for SB, and LPM correction at high energy) but overall I think this is done! Good work and thank you to all!

[sethrj]

Thanks everyone for adding a process. I've created the https://github.com/celeritas-project/celeritas/tree/physics-hackathon branch with templates for the different interactors and associated classes. See you tomorrow!

[sethrj]

Noticed while looking through the Geant4 sampling code that G4eCoulombScatteringModel does actually sample isotopes from an element to choose a nucleus to scatter, so we'll need to implement isotopes/nuclide to complete muon physics.

[whokion]

Again, what is the motivation of adding muon processes/models in the first pass of the project? There are only a few
muons produced by typical HEP events and will be no much work for a GPU task. Are there any other special applications with a large number of muons for targeting a offload GPU task?

Regarding to the isotope, it is still worthwhile to implement anyway for 1) completeness of the material classes for general supports 2) more accurate EM models used them

[sethrj]

@whokion Our initial target isn't to offload only a subset of particles onto the GPU. I think both from a code complexity standpoint and a performance standpoint, it's worth our while to implement everything on device. You are correct that muon physics is not the first priority for us; that's why we're focusing on electron/gamma for now. I was just making a note of the need for isotope for future reference. Do any of the other models in the EM target list require isotope sampling?

[whokion]

Isotope information is used mostly in hadronic physics and several high precision EM models (not part of the Geant4 standard EM physics for e+/- and gamma). If we include electron-nucleus and gamma-nucleus process (which is hadrnic physics, but part of standard EM physics), we definitely need to add Isotope into the material class (which was the reason that I mentioned to add Isotope in the beginnig of material discussion). Personal, I am also interested in exploring neutron physics for GPU in the next stage, so adding the isotope is definitely useful for either near-term and long-term developments.

The reason that I keep arguing to remove muon physics from our scope is not because of the simplicity of codes, but because of performance. If we do not have enough input population of muons for GPU, adding them to the GPU task will not bring any benefit to overall performance (may be even add advert effects). If you are aiming to make celetitas a complete EM physics on GPU, I will leave the decision to you.

[sethrj]

OK, the part about the hadronic EM physics makes sense. You're right that it's definitely worthwhile to add isotopic information (and it will be easy to do).

I think we may be talking past each other with regards to the muon physics. My line of thought is thus:

• Our eventual moonshot goal is to implement all the physics and functionality needed for LHC detector simulation. The first major subset of this goal is EM physics. Among the EM physics processes used for LHC modeling are muons.
• We have to implement the necessary physics kernels in Celeritas, or (if we want CPU-only code) we have to write potentially expensive adapters to Geant4 and allow use of its physics at the same time as ours. I don't think writing a Celeritas->Geant4 adapter is viable at this time.
• The way our Interactor classes are currently written allows GPU-enabled code to be written basically as easily as CPU-enabled code, so there's little motivation to write CPU-only physics kernels in Celeritas.
• The computational overhead for shuttling secondaries back and forth between CPU and GPU will be non-negligible, so there's motivation for keeping all the data on the GPU even if some part of it will be a bit slower.
• The code complexity overhead of maintaining a parallel set of data structures on the CPU, and managing the transfer of secondaries back and forth depending on whether the processes are on CPU or GPU, will be substantial.
• There should be a number of ways we can mitigate the cost of lots of parallel physics kernels if the existence of additional muon physics processes proves to be a problem -- we might be able to launch them all simultaneously, or use GPU parallel reductions to decide that some particle types (and thus some processes) don't need to be launched, or lump multiple Interactor classes together.

Am I wrong about the first bullet point? Or is there something else I'm missing here?

[whokion]

The first bullet point is right. HEP simulation (and the Geant4 standard EM physics) definitely include muons. Nonetheless, my argument is that the CPU consumption or contribution by muon is really a small fraction (sub % level in typical HEP event simulations) and will not impact any performance gain even the celeritas fully supports it. Only relevant source of secondary muons is from the pair production process above the two-muon energy threshold (of which the production probability is very small even in the region) and can be simply transfer to the host.

Regarding to your second bullet, I have a different opinion. The integration of celeritas to Geant4 is easy (actually the current celeritas already has the connectivity to Geant4 and vise versa as you know- i.e, a Geant4 application can use celeritas codes (both host codes and kernels with cuda codes) and I believe that the celeritas host code can call Geant4 functions too at this moment. As the major source of (secondary) electrons and gamma are from the hadronic interactions, there will not enough source of input EM particles in realistic simulation scenarios if we do not integrate celeritas into the Geant4 simulation workflow - as the number of primary electrons and gamma are just handful even multiple events are combined together. One of my interests is eventually integrating celeritas into the Geant4 "task" model which provides input EM particles (i.e, primary and secondary electrons and gamma) from the actual tracking loop which is a real demonstrator of a hybrid computing model on HPC for realistic HEP event simulation.

[sethrj]

OK, so that makes it clearer. It still seems like you're still assuming the hadrons are going to be transported on CPU? I know that's been a pretty popular assumption/suggestion that dates back at least to J. Madsen's performance analyses, but (at least initially) the design goal of Celeritas is to implement everything (except for memory management and I/O) on device.

Our Geant4 integration is quite primitive. Stefano has gotten us the ability to set up the Geant4 data structures and read a small subset of initialization-time data from them, which we then read into our own data structures. There's an enormous gap between that and correctly/robustly getting Geant4 processes/models integrated into a hypothetical Celeritas CPU event loop. The Geant4 models require numerous data structures to be passed through a dizzying hierarchy of classes, and they usually store their state in several different locations in the code (using side effects for the classes) so it's hard to guarantee correctness, the same way it was hard to make Geant4 thread-safe.

That being said, I think it will be an easier task to integrate Celeritas as a module of Geant4 for your vision of it being an accelerator for EM processes, because our capability is a small subset of Geant4's and we're being more careful about separating state information. So I think that could be a more realistic possibility :)

[whokion]

Exactly! One of my interests is to provide a hybrid Geant4 application that offloads EM particle transport on GPU using the celeritas as an external package and simulates the rest of simulation (i.e., hadronic physics) on the host CPU with Geant4-multi-threading task model assuming that HPC systems will be consist of nodes with 1-CPU host and N-devices. I have no problem with the vision that celeritas eventually does everything on device and even supports hadronic physics on GPU (i.e. replacing Geant4), but have a number of arguments that the hybrid computing model may be the better solution for HEP-like detector simulation (even for the longer term) - I will skip lengthy arguments why it is the case here.

Regarding to the integration into Geant4, there should be no much requirements from the celeritas side - The Geant4 task model should provide interfaces to use celeritas kernels or modules to offload special groups of collected particles from its tracking loop and take celeritas outputs (special groups of secondaries and/or hit information) back to Geant4 - I am doing this reverse way for now (i.e., creating a geant4 application within in the celeritas framework). The celeritas design based on the Geant4 importer and related celeritas data structure for GPU has no conflict with the envisaged integration route.

One of related questions that HEP people may ask if the celeritas aims to do everything on devices will be how to supply enough input (electrons or gamma for an example) for a realistic HEP simulation and maintain high occupancy as the self-contained EM cascade will die out relatively very quickly.

[sethrj]

OK, I understand now. Thanks for the explanation! I didn't realize you had plans to write a "celeritas-as-a-geant4-accelerator" app directly inside celeritas. That's a good idea.

As to the final question, I think we'll be running numerous events simultaneously, preferably buffering extra secondaries to the GPU but possibly also to the CPU memory (and if we're really in dire straits, assuming there's a fast NVRAM-type device like Summit has). I don't doubt there will be quite a bit of playing around to find the balance of keeping the device pretty full but not way too full.

[sethrj]

@whokion I think the Rayleigh and positron annihilation "Process" wrapper classes still have to be implemented. It shouldn't be difficult, but since you implemented their "model" counterparts could you take a crack at those before diving too deeply into the next magnetic field capabilities? It's a necessary step for the demo loop #54 .