Numerical Relativity: Paving the way for Gravitational Wave Astronomy

• Deirdre Shoemaker (University of Texas at Austin)

Room: ECJ 1.204 As we embark on a week dedicated to the ETK, I will discuss the role of numerical relativity in gravitational wave astronomy. I hope to motivate future work in NR that is essential for achieving the science discoveries promised by current and future gravitational wave observatories.

(virtual) Introduction to the Einstein Toolkit

• Yosef Zlochower (Rochester Institute of Technology)

Room: ECJ 1.204 This talk provides a broad overview of the Einstein Toolkit, its design, and history, as well as its main features.

Using Cauchy Characteristic Evolution to obtain accurate waveforms at infinity

• Deborah Ferguson (University of Illinois Urbana-Champaign)

Room: ECJ 1.204 One of the primary applications of numerical relativity is the generation of waveforms to be compared against gravitational-wave data. Such a comparison enables us to understand the systems emitting the gravitational waves. As gravitational-wave observatories improve, it's becoming increasingly important that our waveform templates are sufficiently accurate. There are many factors that go into the accuracy of waveforms, but this talk will focus on the challenge of obtaining waveforms at infinity. While several techniques exist for extrapolating waveforms extracted at finite radius out to infinity, every method introduces potential systematic errors. One method that has proved to be quite promising is Cauchy Characteristic Evolution. This talk will introduce CCE_Export, a thorn for exporting Einstein Toolkit data into a format readable by SpECTRE's CCE module. I will also highlight the results of using this method including showing the recovery of memory and comparing to other extrapolation methods and waveforms.

Numerical Relativity in modified gravity

• Helvi Witek (University of Illinois Urbana-Champaign)

Room: ECJ 1.204

Hands-On with the Einstein Toolkit

• Steven Brandt (Louisiana State University)

Room: ECJ 1.204 In this tutorial, students will become familiar with how to download, compile, and run the Einstein Toolkit. If time allows, we may also discuss ways in which students can create their own thorns. The tutorial will be Jupyter notebook-based, so there is no need for students to prepare ahead of time by installing any prerequisites. The only thing students will need is a computer with a functioning web browser. The tutorial server provided for the workshop will continue to operate for a week or so after the workshop, providing any students with ample time to experiment.

Speeding up BBH simulations in the Einstein Toolkit: Multi-Step Runge-Kutta time integrators and Multipatch methods.

• Lucas Timotheo Sanches (Louisiana State University)

Room: ECJ 1.204 In this talk, I will present the latest developments of ongoing efforts at LSU to speed up simulations in the Einstein Toolkit under CarpetX. In particular, I will present our new results using Multi-Step Runge-Kutta time integrators which can speed up BBH evolutions by 30% with simple algorithmic changes. In order to do that, I will carefully review the stability theory of ODE solvers, which should be particularly relevant to new students of Numerical Relativity. I will also present exciting new developments in CapyrX, CarpetX multipatch infrastructure.

(virtual) GPU-accelerated Subcycling Time Integration with the Einstein Toolkit

• Liwei Ji (Rochester Institute of Technology)

Room: ECJ 1.204 Adaptive Mesh Refinement (AMR) with subcycling in time enables different grid levels to advance using their own time steps, ensuring finer grids employ smaller steps for accuracy while coarser grids take larger steps to improve computational efficiency. We present the development, validation, and performance analysis of a subcycling in time algorithm implemented within the CarpetX driver in the Einstein Toolkit framework. This new approach significantly improves upon the previous subcycling implementation in the Carpet driver by achieving higher-order convergence—fourth order in time instead of second order—and enhanced scaling performance. The key innovation lies in optimizing the exchange of ghost points at refinement boundaries, limiting it to the same number as those at inter-process boundaries using dense output from coarser levels, thereby reducing computational and communication overhead compared to the implementation in Carpet, which required a larger number of buffer zones. To validate the algorithm, we first demonstrate its fourth-order convergence using a scalar wave test. We then apply the algorithm to binary black hole (BBH) simulations, confirming its robustness and accuracy in a realistic astrophysical scenario. The results show excellent agreement with the well-established LazEv code. Scaling tests on CPU (Frontera) and GPU (Vista) clusters reveal significant performance gains, with the new implementation achieving improved speed and scalability compared to the Carpet-based version.

(virtual) AsterX: a new open-source GPU-accelerated GRMHD code for dynamical spacetimes

• Jay Kalinani (Rochester Institute of Technology)

Room: ECJ 1.204 With the ongoing transition toward exascale computing to tackle a range of open questions via numerical simulations, the development of GPU-optimized codes has become essential. In this talk, I will highlight the key features of AsterX, a novel open-source, modular, GPU-accelerated general relativistic magnetohydrodynamic (GRMHD) code for fully dynamical spacetimes in 3D Cartesian coordinates. Built for exascale applications, AsterX integrates with CarpetX, the new driver for the Einstein Toolkit, leveraging AMReX for block-structured adaptive mesh refinement (AMR). The code employs the flux-conservative Valencia formulation for GRMHD, and uses high-resolution shock capturing schemes to ensure accurate hydrodynamic modeling. Alongside discussions on the ongoing code development, I will also present the results of comprehensive 1D, 2D, and 3D GRMHD tests conducted on OLCF's Frontier supercomputer, highlighting AsterX's performance gains through subcycling in time and demonstrating its scaling efficiency across thousands of nodes.

EmitCactus: Generating CarpetX Thorns From a Python-Based DSL & CottonmouthBSSN: A New, Fast CarpetX BSSN Code

• Lucas Timotheo Sanches (Louisiana State University)
• Max Morris (Louisiana State University)

Room: ECJ 1.204 In this talk, we will introduce EmitCactus, a development environment which provides users with the capability of generating complete, performant, GPU-ready CarpetX thorns from a high-level, symbolic Python-based DSL. We will discuss the modular design of EmitCactus which allows for the development of custom frontends (e.g., NRPy LaTeX) and backends (e.g., drivers besides CarpetX). We will also introduce CottonmouthBSSN, a new CarpetX BSSN code created from scratch in EmitCactus, comparing its results and performance against CanudaX, a handwritten CarpetX port of the well established Canuda BSSN code. We demonstrate that Cottonmouth is capable of producing comparable results at higher performance with much lower implementation complexity.

EmitCactus Live Demo

• Max Morris (Louisiana State University)
• Lucas Timotheo Sanches (Louisiana State University)

Room: ECJ 1.204 In this talk, we will give a live demo instructing those of all experience levels in the use of EmitCactus. We will guide attendees through the process of writing a recipe in the EmitCactus DSL to generate a GPU-ready CarpetX thorn.

GRHayL — An Open-Source, Modular, and Extensible Library for GRMHD Simulations

• Leonardo Werneck (University of Idaho)

Room: ECJ 1.204 The advent of gravitational-wave (GW) astronomy has revolutionized our understanding of the universe and expanded the frontiers of multi-messenger astrophysics. Accurately interpreting these signals, particularly from phenomena such as binary neutron star mergers, requires detailed numerical simulations that solve the coupled Einstein and general relativistic magnetohydrodynamics (GRMHD) equations. These simulations must incorporate complex physics, including state-of-the-art nuclear equations of state, magnetic fields, and neutrino transport, to capture the interplay between strong gravity, dense matter, and radiation. However, developing and maintaining comprehensive GRMHD codes is a formidable task, demanding significant computational and software engineering effort. In this talk, I will introduce GRHayL — a new, open-source, infrastructure-agnostic library designed to address these challenges. GRHayL delivers modular, extensible implementations of core GRMHD algorithms, optimized for modern heterogeneous architectures with GPU-friendly kernels. By providing a flexible and efficient foundation, GRHayL aims to accelerate the development of next-generation GRMHD codes, reduce computational costs, and facilitate broader access to cutting-edge simulation capabilities within the astrophysics community.

(virtual) Numerical Relativity with Nmesh

• Wolfgang Tichy (Florida Atlantic University)

Room: ECJ 1.204 We give an introduction to the Nmesh code, which is intended to run efficiently on large supercomputers. The goal is to solve challenging relativistic astrophysics problems such as binary neutron star or black hole mergers. To treat matter (e.g. neutron stars) we have implemented the Valencia formulation of the evolution equations for general relativistic hydrodynamics. These can be coupled to the evolution equations for gravity, for which we currently use the Generalized Harmonic system. The principal spatial discretization used in Nmesh is based on a discontinuous Galerkin method. However, we can also switch to finite difference or some form of finite volume methods, when needed in certain regions. We explain these methods and also how we can use a smoothness indicator to decide which scheme to use. We also show first results of the evolution of neutron stars, for single star and binary star cases.

Neutrino Transport in Binary Neutron Star Mergers with AthenaK

• Maitraya Bhattacharyya (Pennsylvania State University)

Room: ECJ 1.204 Binary neutron star mergers (BNS) are extraordinary astrophysical events, acting as an important source for the production of a significant fraction of the universe's heavy r-process elements. Neutrinos drive the composition of the ejected matter and play a crucial role in cooling the massive hot remnant formed from the merger of neutron stars, necessitating their accurate modeling in BNS simulations. In this talk, I will describe our efforts to introduce neutrino physics in the numerical code AthenaK. More specifically, I will introduce two approaches -- an approximate moment-based approach (M1) and a full Boltzmann treatment with filtered spherical harmonics (FP_N) and finite element methods on geodesic grids (FEM_N), discussing the challenges in solving the full seven-dimensional problem and limitations of the M1 approach. Finally, I introduce a new library for neutrino-matter interaction for use on GPUs.

Performance-Portable Numerical Relativity with AthenaK

• Hengrui Zhu (Princeton University)

Room: ECJ 1.204 In this talk, I present the numerical relativity module developed within AthenaK, an open-source, performance-portable astrophysics code optimized for exascale computing. Driven by the demand for high-accuracy gravitational waveforms and the need to efficiently utilize emerging hardware architectures, AthenaK adopts the Z4c formulation to evolve the Einstein equations. We demonstrate the accuracy of the implementation through a suite of standard tests, including convergence studies of gravitational waveforms from binary black hole mergers. We will also discuss the implementation of Valencia GRMHD, with validation tests including binary neutron star evolutions.

(virtual) The Guided Moments Formalism

• Manuel Izquierdo (Universitat de les Illes Balears)

Room: ECJ 1.204 Accurate modeling of neutrino transport plays a crucial role in understanding astrophysical phenomena such as core-collapse supernovae and neutron star mergers. In this seminar we will review two popular methods for approximating the seven-dimensional Boltzmann equation: the truncated momentum formalism (M1 scheme) and Monte-Carlo (MC) algorithms. Then we present the Guided Moment (GM) formalism, which combines efficiently both methods to capture accurately both optically thick and thin limits. A comparison between the three schemes (GM, M1, MC) will demonstrate the tremendous potential of the GM formalism.

(virtual) Low-level tricks and high-level vision for performant neutrino transport

• Jonah Miller (Los Alamos National Laboratory)

Room: ECJ 1.204 As the numerical relativity community focuses on detailed microphysics in neutron star mergers, and as accelerator hardware becomes ubiquitous, GPU-resident simulations and neutrino transport are becoming routine. I discuss my experience with both performance portability and Monte Carlo neutrino transport for post-neutron star merger disk and kilonova modeling, with an emphasis on gotchas and lessons learned. My goal is to provide useful information to both new members of the ET community and the core development team as they move towards implementing transport in community codes. I also speak to exciting developments in the field, such as in-situ treatment of neutrino oscillations and tensor-network based radiation transport.

BHaHAHA: A Fast, Robust Apparent Horizon Finder Library using Hyperbolic Relaxation

• Zachariah Etienne (University of Idaho)

Room: ECJ 1.204 We present BHaHAHA (BlackHoles@Home Apparent Horizon Algorithm), a new open-source apparent horizon (AH) finder that uses hyperbolic relaxation, reformulating the elliptic Marginally Outer Trapped Surface (MOTS) equation as a damped scalar wave equation on the 2-sphere via a reference-metric approach. As such, it exists as the first-ever hyperbolic flow method. Key techniques—such as multigrid-inspired refinement, over-relaxation, OpenMP parallelization, and dynamic tracking optimizations (e.g., extrapolated initial guesses, reduced-volume interpolation)—make BHaHAHA both efficient and robust. Benchmarks show BHaHAHA is scale-invariant across 16 orders of magnitude in mass (where AHFinderDirect struggles), and 2.1x faster in dynamic scenarios like GW150914 while maintaining similar accuracy (relative area errors ~1e-5). It also reproduces high-precision results in difficult cases like a q=4 common horizon and a 3-BH Brill-Lindquist critical radius. BHaHAHA is infrastructure-agnostic and has been integrated with both the Einstein Toolkit and BlackHoles@Home frameworks so far, with more infrastructure implementations in development.

SPHINCS-BSSN: Relativistic Hydrodynamics Using Particles

• Peter Diener (Louisiana State University)

Room: ECJ 1.204 I will describe SPHINCS-BSSN: a code that combines spacetime evolution on a grid with a Lagrangian treatment of the matter. That is, the matter is evolved using the method of Smoothed Particle Hydrodynamics (SPH). I will describe the advantages (and disadvantages) of using particle methods and will describe how we overcame the technical challenges of coupling the matter to the spacetime. I will finally present some basic tests of the code as well as some new results.

Neutrino Radiation Hydrodynamics with Flash-X+thornado

• Vassilios Mewes (Oak Ridge National Laboratory)

Room: ECJ 1.204 Neutrino-matter coupling via weak interactions is one of the most important physical mechanisms in the evolution of core-collapse supernovae (CCSN). The numerical modeling of these systems is an inherently multi-physics, multi-method and multi-scale problem. In this talk, we will give an introduction to neutrino radiation hydrodynamics and then describe the union of three codes to simulate CCSN: Flash-X+thornado+WeakLib, which evolve the fluid and gravity and provide the computational infrastructure (Flash-X), a code that evolves neutrino radiation hydrodynamics in a spectral two-moment model using the discontinuous Galerkin method (thornado), and a library that provides the equation of state and weak interaction opacity tables (WeakLib). We will describe the two-moment spectral neutrino transport in thornado, the mapping of fluid data in different representations (finite volume and discontinuous Galerkin), as well as our performance portability strategy using OpenMP/OpenACC offloading to harness heterogeneous exascale machines such as Frontier at the OLCF. We will also present the Oak Ridge Leadership Computing Facility at Oak Ridge National Laboratory and the computing opportunities for researchers through programs like DOE INCITE.

(virtual) Dendro-GR: Scalable wavelet-based AMR for numerical relativity

• Milinda Fernando (University of Texas at Austin)

Room: ECJ 1.204 Simulations to calculate a single gravitational waveform (GW) can take several weeks. Yet, thousands of such simulations are needed to detect and interpret gravitational waves. Future detectors will require even more accurate waveforms than those currently used. Here, we discuss the Dendro-GR framework, a large-scale, wavelet-driven octree-based adaptive mesh refinement with support for multi-GPU acceleration with performance analysis and benchmarking. We achieve 800 GFlops/s on a single NVIDIA A100 GPU with an overall 2.5x speedup over a two-socket, 128-core AMD EPYC 7763 CPU node with an equivalent CPU implementation. We present detailed performance analyses, parallel scalability results, and accuracy assessments for GWs computed for mass ratios q=1,2,4. We also present strong scalability up to 16 A100s and weak scaling up to 229,376 x86 cores on the Texas Advanced Computing Center's Frontera system.