Delta tracking#3971
Conversation
This reverts commit ab346b2.
|
Looping in @realmisch on this PR for awareness. |
Additional Production TestingMany thanks to the folks who provided models for testing! All models are run with 128 threads of a single node of Improv @ ANL. Gas-Cooled Microreactor (model provided by Lewis Gross at UW-Madison)This model is a TRISO-fueled (40% packing fraction) prismatic HTGR with gadolinium poison particles (25% packing fraction), boron carbide control drums, and boron carbide control rods. These absorbers inflate the majorant in the thermal portion of the spectrum, resulting in reduced delta tracking performance. We run the model with 10000 particles per batch, 100 inactive batches, and 1000 active batches. I believe the reduced inactive rate (compared to the active rate) is caused by the building of cell neighbourhood lists in inactive batches, which is substantially slower than tallying in this problem due to the TRISO/poison particle surfaces.
BEAVRSThis model is a prototypic whole-core PWR using oxide fuel of varying levels of enrichment. Absorbers include borosilicate glass, boron carbide control rods, and cadmium control rods. In an identical manner to the GCMR, these absorbers inflate the majorant in the thermal region (just not to the same degree as the poison particles). We run the model with 1000000 particles per batch, 100 inactive batches, and 1000 active batches.
ARC-100 (model provided by Nathan Glaser at OSU)This model is a prototypic small SFR using metallic fuel of various levels of enrichment. Boron carbide control rods are the only absorbers in the geometry. The vast majority of neutrons in this SFR transport above energies where the inflated majorant degrades performance, and so we see a fairly significant speedup when running delta tracking. We run the model with 1000000 particles per batch, 100 inactive batches, and 1000 active batches.
TakeawaysDelta tracking (as implemented in this PR) can improve performance for certain whole-core systems which are not susceptible to inflated majorant cross sections, namely fast reactors. We anticipate that hybrid delta tracking (as implemented in Serpent) will mitigate the observed performance degradations in thermal systems. A first cut of hybrid tracking in OpenMC based on this PR shows promise, where we see decent speedups for thermal problems with strong absorbers (results to come in the hybrid tracking PR). 😄 |
2 participants
Description
This pull request implements standard delta tracking in OpenMC for the continuous-energy Monte Carlo solver. This includes history-based mode, event-based mode, and photon transport. Both the history-based and event-based implementations support the shared secondary bank recently implemented in #3863.
Neutron majorants are generated in a two-step process. The energy grids for all nuclides (including both smooth cross sections and unresolved resonance probability tables) in the problem are unionized. Afterwards, the total cross section for each nuclide is interpolated to the unionized grid to facilitate the computation of a per-material maximum cross section (the maximum value of which is used for the global majorant cross section). Thermal scattering laws are applied for materials which specify them. Unresolved resonance probability tables are treated in a conservative manner, where the maximum value over all bands is computed and then interpolated onto the unionized grid. Temperatures are handled in a conservative manner, where the majorant is the maximum over the entire temperature grid. Distributed cell densities are applied by finding the maximum density multiplier over the domain for each material, caching the result, and using the corresponding maximum density multiplier to adjust the per-material maximum cross section. Photon majorants are generated in a similar manner, but their computation is more straight-forward. The smooth cross section grids of all required photon interactions are unionized, and cross sections are then computed on the new unionized grid to compute the majorant.
The delta tracking history-based algorithm itself is quite simple. Particles are advanced by the smallest of either the distance to next collision (sampled from the majorant cross section) or the distance to the nearest external boundary surface. At that point, the particles are located by a call to global point containment. If the particle is crossing a boundary surface, the BC is applied and the particle goes back to advancing (if its alive). Otherwise, cross sections at the real/virtual collision point are computed and used to rejection sample the majorant. If the collision is virtual, the particle goes back to advancing. Otherwise, the collision is processed.
The event-based algorithm essentially implements this process verbatim with the current event queues and different logic to shuffle particles between queues. This is great in that it yields identical particle histories between approaches (which can be seen in the testing results), but is almost certainly leaving performance on the table. The event-based implementation is provided for feature parity only, and we leave performance improvements in this part of delta tracking to future work.
Limitations
This PR does not support multigroup mode (under development by folks at LLNL, see #3902). Additionally, windowed multi-pole cross sections are not currently supported, nor are NCrystal materials. The usual delta tracking limitation of collision estimators is also enforced in this PR.
Testing
Regression tests are provided for history-based delta tracking, event-based delta tracking, distributed cell densities, and all boundary conditions. The first three tests focus on the majorant calculation (to ensure it hits features that impact cross section generation), and the last test ensures boundary conditions function as intended. I intended to add a temperature interpolation test, but none of the nuclides in the NNDC library appear to have temperature interpolation tables. I was also considering adding a test for distributed materials and the shared secondary bank algorithm, but decided against it for the initial PR due to the large size. I am more than happy to add more tests if people are fine with reviewing them. 😃
Verification & Initial Performance Results
Initial verification and performance results can be found in this delta tracking benchmark repo, alongside the models used. The following systems are modelled:
All simulations are run in history-based mode in both neutron-only and neutron+photon coupled transport. The cases use 1000, 10000, or 100000 particles per batch with 100 inactive batches and 1100 total batches. Several quantities are tallied. These include neutron & photon spectra (101 energy bins), total reaction rates and particle fluxes (structured mesh tallies with 51x51 bins in the x-y plane) that filter by particle type, and global quantities (leakage fraction and$k_{eff}$ with the collision estimator). It should be noted that the LWR, SFR, and Jezebel problems do not contain surface subdivisions (the best-case scenario for surface tracking). All problems are run on a single node of Improv @ ANL (128 available OpenMP threads).
Integral Quantities
We start by comparing the convergence of integral quantities. Surface tracking and delta tracking agree within$3\sigma$ (the error bounds in the plots) for both leakage fractions and $k_{eff}$ , which is exactly what we want to see! This gives confidence that the implementation of both reflective and vacuum boundary conditions is correct, and the majorant is conservative. We do not hit the rejection sampling guard for any of these problems even after simulating ~ $10^8$ particle histories.
Spectra
The neutron spectra (computed with single-physics runs to improve statistics) and photon spectra can be found below for the 100000 particle/batch case. The agreement for all cases is within$3\sigma$ .
In addition to the raw spectra, the$L_2$ difference between the tracking methods is plotted below. We'd expect to see that the difference follows a $1/\sqrt{N}$ trend with respect to the number of particles per batch, which would indicate that delta tracking and surface tracking are converging to the same values. We're close to the expected $1/\sqrt{N}$ convergence rate, with some noise due to the low number of samples in the ultra thermal range.
Spatial Distributions
It is difficult to visualize differences between the spatial distributions, and so only the$L_{2}$ differences in the total reaction rates are plotted. Once again, the expected $1/\sqrt{N}$ rate of convergence is observed for both the neutron and photon reaction rates!
Performance
We compare the performance of delta tracking to surface tracking by defining the following speedup:$S = R_{delta}/R_{surface}$ . $R_{delta}$ is the active tracking rate with delta tracking, and $R_{surface}$ is the active tracking rate with surface tracking. We see modest speedups when running delta tracking in Jezebel and the pincell problems (SFR & LWR). The expected performance improvement materializes when running the HTGR compact problem, where delta tracking is close to 5x faster than surface tracking!
Future Work
This is only an initial implementation of delta tracking. Next steps will include the implementation of hybrid tracking (in a manner similar to Serpent) to mitigate performance degradations that are commonly observed when using delta tracking in problems with localized absorbers. I'm also looking at improving cell containment routines using novel data structures. Performance benchmarking is ongoing with production models (whole core problems). I am also quite interested in optimizing the event-based delta tracking approach to pave the way for a GPU implementation of the method.
Closes #24, closes #3342, closes #3343
Checklist