1625 - Addressing Rare Outcomes in PER Quantitative Studies

Author / Creator

AAPT WM21 (2021)

Conferences

AAPT WM21 C1.6 Physics Education Research II Posters (2021)

Available as

Online

Summary

We encounter variables with little variation often in PER due to the demographics of physics and the questions we ask. For example, in course completion studies, most students will earn high enough...

We encounter variables with little variation often in PER due to the demographics of physics and the questions we ask. For example, in course completion studies, most students will earn high enough grades to pass the course. Yet, little work has examined how to analyze such data. Therefore, we conducted a simulation study using logistic regression, penalized regression, and random forest. We systematically varied the fraction of positive outcomes, feature imbalances, and odds ratios. We find the algorithms treat features with the same odds ratios differently based on their imbalance and the outcome imbalance. While none of the algorithms solved the problem, some reduced the scale of the problem. Our results suggest that PER studies may contain false negatives when determining which variables are related to an outcome. We propose recommendations for researchers and then illustrate these by predicting which applicants will be admitted to a graduate physics program.

Show More