26th Australian Institute of Physics Congress and the Australian and New Zealand Conference on Optics and Photonics

Comparative Analysis of Machine Learning Models for Early Prediction of Student Performance Using LMS Log Data

Not scheduled

20m

Belinda Hutchinson Building (The University of Sydney )

Poster AIP | Physics Education (PEG)

Description

Identifying students at risk of academic failure early in the semester is critical for timely intervention. Previous work in physics education has shown that institutional variables such as cumulative GPA are critical for early prediction, with in-class variables becoming important only after the first examination [1], and that tuning the model was required to achieve better performance when predicting unbalanced outcomes [2]. While LMS-based early prediction has been demonstrated across multiple disciplines [3], its application to physics education remains largely unexplored. This study compares multiple machine learning models for predicting student pass/fail outcomes using first four weeks of LMS log data from a physics course, and evaluates the robustness of the best-performing model.
We analyzed LMS log data from 621 students, extracting seven behavioral features: four features capturing the number and type of interactions (total number of log events, number of assignment interactions, number of content views, and number of forum actions), and three calculated features that describe the temporal patterns of the data (total number of unique days, percentage of events on weekends, and percentage of events between 10pm and 4am). Seven models were evaluated: Logistic Regression, Random Forest, Gradient Boosting, Support Vector Machine, K-Nearest Neighbors, Decision Tree, and XGBoost. All models underwent hyperparameter optimization using randomized and grid search with 5-fold stratified cross-validation.
Initial model comparison showed Decision Tree achieving the highest balanced accuracy (72.3%). After extensive tuning, XGBoost emerged as the superior model, achieving a balanced accuracy of 74.1%. XGBoost successfully identified 89.7% of students who would fail while maintaining 88.5% precision when predicting failure. Robustness testing revealed consistent performance across all train-test splits (balanced accuracy range: 72.0–74.3%), with stratified cross-validation yielding highly stable results (74.2–74.6%). This work demonstrates a proof of principle that XGBoost can identify at-risk students, enabling physics educators to proactively intervene.