Speaker
Description
Research into foundation models for high energy physics has moved fast beyond the initial prototyping efforts. As the community has been exploring input representations, pretraining strategies and downstream tasks, the number of models has skyrocketed. This talk will not give an overview of all those models; rather, I aim to give an overview of the underlying methods. From next token prediction and masked modeling to supervised learning and joint-embedding approaches, different pretraining tasks result in different learned representations, some of which may be better aligned with certain downstream tasks than others. I will also explore what it would take for the field to actually start using foundation models in a real analysis, a question that leads to a piece that has so far been missing in this story: evaluation.