2025 North American Einstein Toolkit Workshop

9 Jun 2025, 08:00 → 13 Jun 2025, 17:00 America/Chicago

ECJ 1.204 (University of Texas at Austin)

We are pleased to announce the 2025 edition of the North American Einstein Toolkit Workshop. The workshop will be held at the University of Texas at Austin. This is an opportunity for researchers and students to learn about the Einstein Toolkit (https://einsteintoolkit.org/), a community-driven software platform of core computational tools to advance and support research in relativistic astrophysics and gravitational physics.

The workshop will offer a mixture of talks and tutorials. The tutorials provide an opportunity for new users to become familiar with the Einstein Toolkit cyberinfrastructure. The talks are aimed at highlighting exciting science using the tools of numerical relativity, in particular, the Einstein Toolkit. At the end of the workshop, there will be a broad discussion on future directions of the Einstein Toolkit.

Zoom: https://utexas.zoom.us/j/95794003532

Registration: Registration has ended.

Travel support: Limited financial support is available for attendees. This can be requested in the registration form. To receive full consideration, requests must be submitted by April 30.

Lodging: A list of recommended hotels can be found here.

Sponsored by:  National Science Foundation: Frameworks: E=mc2 - Enabling Multimessenger astrophysics through Community-driven Cyberinfrastructure (119025-20128). Travel support provided by OAC-2004879, OAC-2004157.

Monday 9 June

Check-in

Welcome

1 Numerical Relativity: Paving the way for Gravitational Wave Astronomy

As we embark on a week dedicated to the ETK, I will discuss the role of numerical relativity in gravitational wave astronomy. I hope to motivate future work in NR that is essential for achieving the science discoveries promised by current and future gravitational wave observatories.

Speaker: Deirdre Shoemaker (University of Texas at Austin)

Coffee Break

2 (virtual) Introduction to the Einstein Toolkit

This talk provides a broad overview of the Einstein Toolkit, its design, and history, as well as its main features.

Speaker: Yosef Zlochower (Rochester Institute of Technology)

3 Using Cauchy Characteristic Evolution to obtain accurate waveforms at infinity

One of the primary applications of numerical relativity is the generation of waveforms to be compared against gravitational-wave data. Such a comparison enables us to understand the systems emitting the gravitational waves. As gravitational-wave observatories improve, it's becoming increasingly important that our waveform templates are sufficiently accurate. There are many factors that go into the accuracy of waveforms, but this talk will focus on the challenge of obtaining waveforms at infinity. While several techniques exist for extrapolating waveforms extracted at finite radius out to infinity, every method introduces potential systematic errors. One method that has proved to be quite promising is Cauchy Characteristic Evolution. This talk will introduce CCE_Export, a thorn for exporting Einstein Toolkit data into a format readable by SpECTRE's CCE module. I will also highlight the results of using this method including showing the recovery of memory and comparing to other extrapolation methods and waveforms.

Speaker: Deborah Ferguson (University of Illinois Urbana-Champaign)

Lunch

4 Numerical Relativity in modified gravity

Speaker: Helvi Witek (University of Illinois Urbana-Champaign)

Coffee Break

5 Hands-On with the Einstein Toolkit

In this tutorial, students will become familiar with how to download, compile, and run the Einstein Toolkit. If time allows, we may also discuss ways in which students can create their own thorns. The tutorial will be Jupyter notebook-based, so there is no need for students to prepare ahead of time by installing any prerequisites. The only thing students will need is a computer with a functioning web browser. The tutorial server provided for the workshop will continue to operate for a week or so after the workshop, providing any students with ample time to experiment.

Speaker: Steven Brandt (Louisiana State University)

Tuesday 10 June

6 Speeding up BBH simulations in the Einstein Toolkit: Multi-Step Runge-Kutta time integrators and Multipatch methods.

In this talk, I will present the latest developments of ongoing efforts at LSU to speed up simulations in the Einstein Toolkit under CarpetX. In particular, I will present our new results using Multi-Step Runge-Kutta time integrators which can speed up BBH evolutions by 30% with simple algorithmic changes. In order to do that, I will carefully review the stability theory of ODE solvers, which should be particularly relevant to new students of Numerical Relativity. I will also present exciting new developments in CapyrX, CarpetX multipatch infrastructure.

Speaker: Lucas Timotheo Sanches (Louisiana State University)

Coffee Break

7 (virtual) GPU-accelerated Subcycling Time Integration with the Einstein Toolkit

Adaptive Mesh Refinement (AMR) with subcycling in time enables different grid levels to advance using their own time steps, ensuring finer grids employ smaller steps for accuracy while coarser grids take larger steps to improve computational efficiency. We present the development, validation, and performance analysis of a subcycling in time algorithm implemented within the CarpetX driver in the Einstein Toolkit framework. This new approach significantly improves upon the previous subcycling implementation in the Carpet driver by achieving higher-order convergence—fourth order in time instead of second order—and enhanced scaling performance. The key innovation lies in optimizing the exchange of ghost points at refinement boundaries, limiting it to the same number as those at inter-process boundaries using dense output from coarser levels, thereby reducing computational and communication overhead compared to the implementation in Carpet, which required a larger number of buffer zones. To validate the algorithm, we first demonstrate its fourth-order convergence using a scalar wave test. We then apply the algorithm to binary black hole (BBH) simulations, confirming its robustness and accuracy in a realistic astrophysical scenario. The results show excellent agreement with the well-established LazEv code. Scaling tests on CPU (Frontera) and GPU (Vista) clusters reveal significant performance gains, with the new implementation achieving improved speed and scalability compared to the Carpet-based version.

Speaker: Liwei Ji (Rochester Institute of Technology)

8 (virtual) AsterX: a new open-source GPU-accelerated GRMHD code for dynamical spacetimes

With the ongoing transition toward exascale computing to tackle a range of open questions via numerical simulations, the development of GPU-optimized codes has become essential. In this talk, I will highlight the key features of AsterX, a novel open-source, modular, GPU-accelerated general relativistic magnetohydrodynamic (GRMHD) code for fully dynamical spacetimes in 3D Cartesian coordinates. Built for exascale applications, AsterX integrates with CarpetX, the new driver for the Einstein Toolkit, leveraging AMReX for block-structured adaptive mesh refinement (AMR). The code employs the flux-conservative Valencia formulation for GRMHD, and uses high-resolution shock capturing schemes to ensure accurate hydrodynamic modeling. Alongside discussions on the ongoing code development, I will also present the results of comprehensive 1D, 2D, and 3D GRMHD tests conducted on OLCF's Frontier supercomputer, highlighting AsterX's performance gains through subcycling in time and demonstrating its scaling efficiency across thousands of nodes.

Speaker: Jay Kalinani (Rochester Institute of Technology)

Lunch

9 EmitCactus: Generating CarpetX Thorns From a Python-Based DSL & CottonmouthBSSN: A New, Fast CarpetX BSSN Code

In this talk, we will introduce EmitCactus, a development environment which provides users with the capability of generating complete, performant, GPU-ready CarpetX thorns from a high-level, symbolic Python-based DSL. We will discuss the modular design of EmitCactus which allows for the development of custom frontends (e.g., NRPy LaTeX) and backends (e.g., drivers besides CarpetX). We will also introduce CottonmouthBSSN, a new CarpetX BSSN code created from scratch in EmitCactus, comparing its results and performance against CanudaX, a handwritten CarpetX port of the well established Canuda BSSN code. We demonstrate that Cottonmouth is capable of producing comparable results at higher performance with much lower implementation complexity.

Speakers: Lucas Timotheo Sanches (Louisiana State University), Max Morris (Louisiana State University)

Coffee Break

10 EmitCactus Live Demo

In this talk, we will give a live demo instructing those of all experience levels in the use of EmitCactus. We will guide attendees through the process of writing a recipe in the EmitCactus DSL to generate a GPU-ready CarpetX thorn.

Speakers: Lucas Timotheo Sanches (Louisiana State University), Max Morris (Louisiana State University)

11 GRHayL — An Open-Source, Modular, and Extensible Library for GRMHD Simulations

The advent of gravitational-wave (GW) astronomy has revolutionized our understanding of the universe and expanded the frontiers of multi-messenger astrophysics. Accurately interpreting these signals, particularly from phenomena such as binary neutron star mergers, requires detailed numerical simulations that solve the coupled Einstein and general relativistic magnetohydrodynamics (GRMHD) equations. These simulations must incorporate complex physics, including state-of-the-art nuclear equations of state, magnetic fields, and neutrino transport, to capture the interplay between strong gravity, dense matter, and radiation.

However, developing and maintaining comprehensive GRMHD codes is a formidable task, demanding significant computational and software engineering effort. In this talk, I will introduce GRHayL — a new, open-source, infrastructure-agnostic library designed to address these challenges. GRHayL delivers modular, extensible implementations of core GRMHD algorithms, optimized for modern heterogeneous architectures with GPU-friendly kernels. By providing a flexible and efficient foundation, GRHayL aims to accelerate the development of next-generation GRMHD codes, reduce computational costs, and facilitate broader access to cutting-edge simulation capabilities within the astrophysics community.

Speaker: Leonardo Werneck (University of Idaho)

Wednesday 11 June

12 (virtual) Numerical Relativity with Nmesh

We give an introduction to the Nmesh code, which is intended to run efficiently on large supercomputers. The goal is to solve challenging relativistic astrophysics problems such as binary neutron star or black hole mergers. To treat matter (e.g. neutron stars) we have implemented the Valencia formulation of the evolution equations for general relativistic hydrodynamics. These can be coupled to the evolution equations for gravity, for which we currently use the Generalized Harmonic system. The principal spatial discretization used in Nmesh is based on a discontinuous Galerkin method. However, we can also switch to finite difference or some form of finite volume methods, when needed in certain regions. We explain these methods and also how we can use a smoothness indicator to decide which scheme to use. We also show first results of the evolution of neutron stars, for single star and binary star cases.

Speaker: Wolfgang Tichy (Florida Atlantic University)

Coffee Break

13 Neutrino Transport in Binary Neutron Star Mergers with AthenaK

Binary neutron star mergers (BNS) are extraordinary astrophysical events, acting as an important source for the production of a significant fraction of the universe's heavy r-process elements. Neutrinos drive the composition of the ejected matter and play a crucial role in cooling the massive hot remnant formed from the merger of neutron stars, necessitating their accurate modeling in BNS simulations. In this talk, I will describe our efforts to introduce neutrino physics in the numerical code AthenaK. More specifically, I will introduce two approaches -- an approximate moment-based approach (M1) and a full Boltzmann treatment with filtered spherical harmonics (FP_N) and finite element methods on geodesic grids (FEM_N), discussing the challenges in solving the full seven-dimensional problem and limitations of the M1 approach. Finally, I introduce a new library for neutrino-matter interaction for use on GPUs.

Speaker: Maitraya Bhattacharyya (Pennsylvania State University)

14 Performance-Portable Numerical Relativity with AthenaK

In this talk, I present the numerical relativity module developed within AthenaK, an open-source, performance-portable astrophysics code optimized for exascale computing. Driven by the demand for high-accuracy gravitational waveforms and the need to efficiently utilize emerging hardware architectures, AthenaK adopts the Z4c formulation to evolve the Einstein equations. We demonstrate the accuracy of the implementation through a suite of standard tests, including convergence studies of gravitational waveforms from binary black hole mergers. We will also discuss the implementation of Valencia GRMHD, with validation tests including binary neutron star evolutions.

Speaker: Hengrui Zhu (Princeton University)

Lunch

Thursday 12 June

15 (virtual) The Guided Moments Formalism

Accurate modeling of neutrino transport plays a crucial role in understanding astrophysical phenomena such as core-collapse supernovae and neutron star mergers. In this seminar we will review two popular methods for approximating the seven-dimensional Boltzmann equation: the truncated momentum formalism (M1 scheme) and Monte-Carlo (MC) algorithms. Then we present the Guided Moment (GM) formalism, which combines efficiently both methods to capture accurately both optically thick and thin limits. A comparison between the three schemes (GM, M1, MC) will demonstrate the tremendous potential of the GM formalism.

Speaker: Manuel Izquierdo (Universitat de les Illes Balears)

Coffee Break

Talk: Elias Most

16 (virtual) Low-level tricks and high-level vision for performant neutrino transport

As the numerical relativity community focuses on detailed microphysics in neutron star mergers, and as accelerator hardware becomes ubiquitous, GPU-resident simulations and neutrino transport are becoming routine. I discuss my experience with both performance portability and Monte Carlo neutrino transport for post-neutron star merger disk and kilonova modeling, with an emphasis on gotchas and lessons learned. My goal is to provide useful information to both new members of the ET community and the core development team as they move towards implementing transport in community codes. I also speak to exciting developments in the field, such as in-situ treatment of neutrino oscillations and tensor-network based radiation transport.

Speaker: Jonah Miller (Los Alamos National Laboratory)

Lunch

17 BHaHAHA: A Fast, Robust Apparent Horizon Finder Library using Hyperbolic Relaxation

We present BHaHAHA (BlackHoles@Home Apparent Horizon Algorithm), a new open-source apparent horizon (AH) finder that uses hyperbolic relaxation, reformulating the elliptic Marginally Outer Trapped Surface (MOTS) equation as a damped scalar wave equation on the 2-sphere via a reference-metric approach. As such, it exists as the first-ever hyperbolic flow method. Key techniques—such as multigrid-inspired refinement, over-relaxation, OpenMP parallelization, and dynamic tracking optimizations (e.g., extrapolated initial guesses, reduced-volume interpolation)—make BHaHAHA both efficient and robust. Benchmarks show BHaHAHA is scale-invariant across 16 orders of magnitude in mass (where AHFinderDirect struggles), and 2.1x faster in dynamic scenarios like GW150914 while maintaining similar accuracy (relative area errors ~1e-5). It also reproduces high-precision results in difficult cases like a q=4 common horizon and a 3-BH Brill-Lindquist critical radius. BHaHAHA is infrastructure-agnostic and has been integrated with both the Einstein Toolkit and BlackHoles@Home frameworks so far, with more infrastructure implementations in development.

Speaker: Zachariah Etienne (University of Idaho)

Coffee Break

18 SPHINCS-BSSN: Relativistic Hydrodynamics Using Particles

I will describe SPHINCS-BSSN: a code that combines spacetime evolution on a grid with a Lagrangian treatment of the matter. That is, the matter is evolved using the method of Smoothed Particle Hydrodynamics (SPH). I will describe the advantages (and disadvantages) of using particle methods and will describe how we overcame the technical challenges of coupling the matter to the spacetime. I will finally present some basic tests of the code as well as some new results.

Speaker: Peter Diener (Louisiana State University)

19 Neutrino Radiation Hydrodynamics with Flash-X+thornado

Neutrino-matter coupling via weak interactions is one of the most important physical mechanisms in the evolution of core-collapse supernovae (CCSN). The numerical modeling of these systems is an inherently multi-physics, multi-method and multi-scale problem. In this talk, we will give an introduction to neutrino radiation hydrodynamics and then describe the union of three codes to simulate CCSN: Flash-X+thornado+WeakLib, which evolve the fluid and gravity and provide the computational infrastructure (Flash-X), a code that evolves neutrino radiation hydrodynamics in a spectral two-moment model using the discontinuous Galerkin method (thornado), and a library that provides the equation of state and weak interaction opacity tables (WeakLib). We will describe the two-moment spectral neutrino transport in thornado, the mapping of fluid data in different representations (finite volume and discontinuous Galerkin), as well as our performance portability strategy using OpenMP/OpenACC offloading to harness heterogeneous exascale machines such as Frontier at the OLCF. We will also present the Oak Ridge Leadership Computing Facility at Oak Ridge National Laboratory and the computing opportunities for researchers through programs like DOE INCITE.

Speaker: Vassilios Mewes (Oak Ridge National Laboratory)

Friday 13 June

20 (virtual) Dendro-GR: Scalable wavelet-based AMR for numerical relativity

Simulations to calculate a single gravitational waveform (GW) can take several weeks. Yet, thousands of such simulations are needed to detect and interpret gravitational waves. Future detectors will require even more accurate waveforms than those currently used. Here, we discuss the Dendro-GR framework, a large-scale, wavelet-driven octree-based adaptive mesh refinement with support for multi-GPU acceleration with performance analysis and benchmarking. We achieve 800 GFlops/s on a single NVIDIA A100 GPU with an overall 2.5x speedup over a two-socket, 128-core AMD EPYC 7763 CPU node with an equivalent CPU implementation. We present detailed performance analyses, parallel scalability results, and accuracy assessments for GWs computed for mass ratios q=1,2,4. We also present strong scalability up to 16 A100s and weak scaling up to 229,376 x86 cores on the Texas Advanced Computing Center's Frontera system.

Speaker: Milinda Fernando (University of Texas at Austin)

Coffee Break

Discussion

Concluding remarks