Supercomputer simulates 13.4 trillion dark matter particles

Eight thousand one hundred computing nodes, about 50,000 graphics processing units (GPUs, the chips that handle massively parallel calculations), and four days. That is what it took for researchers at the US Department of Energy's Argonne National Laboratory to complete two of the largest cosmological simulations ever, modeling 13.4 trillion dark matter particles across a 28.6-billion-light-year region of the universe.

The goal is cosmic inflation, a proposed period of extremely rapid expansion shortly after the Big Bang. Inflation stretched tiny quantum fluctuations across vast distances. Over billions of years, gravity amplified these small density variations into the large-scale cosmic web of galaxies and dark matter observed today. Dark matter cannot be seen through ordinary light, so the simulations use particles to trace its density and velocity, which lets scientists model how cosmic structures evolved.

The team, led by Argonne computational scientist Nicholas Frontiere, is looking for primordial non-Gaussianity: subtle departures from the simple statistical distribution predicted by standard inflationary models. Detecting such deviations could help distinguish between competing explanations for inflation. The signal is expected to be faint, and cosmic variance, the statistical uncertainty that arises when observations cover only a finite portion of the universe, can obscure it. Researchers therefore need simulations that combine enormous volumes with enough resolution to capture dark matter halos, the structures in which galaxies form.

Aurora has 10,624 nodes and 63,744 GPUs, with high-capacity memory and a storage system built for huge data flows. The runs generated more than 85 petabytes of data, mostly checkpoints that let a calculation resume after an interruption instead of starting over, and the team kept about 8 petabytes for scientific analysis. The code used, the Hardware/Hybrid Accelerated Cosmology Code (HACC), was built to run large cosmological simulations across different supercomputing architectures. Unlike an earlier project on Oak Ridge National Laboratory's Frontier system, which modeled gravity alongside gas, stars and other processes, the Aurora runs model gravity alone. That choice let the team cover a larger volume with more particles.

Astronomers working with DESI and NASA's SPHEREx mission are the first to gain from this work. The two simulations share the same cosmological model and non-Gaussian signal, but their initial phases are flipped, so analyzing them together cancels some random variation. Their signal strength is hidden, which lets researchers test analysis methods without knowing the answer beforehand. The resulting synthetic observations are meant to validate workflows and identify potential biases before they are applied to real data, as the surveys gather increasingly detailed observations. The project received computing time through the Department of Energy's Innovative and Novel Computational Impact on Theory and Experiment (INCITE) program.