Feature Engineering & Data Preprocessing

Four published studies found that machine learning predicted the onset of civil wars better than the logistic regressions political scientists had relied on for decades, and all four were wrong for reasons that had nothing to do with their models. Sayash Kapoor and Arvind Narayanan of Princeton re-ran them in 2022. The first, a 2016 random-forest study in Political Analysis, had filled in missing values with rfImpute, an R function that imputes from the proximities of a random forest grown with the outcome itself, and had run it on the training and test rows together, so the test rows were completed with help from the test labels. Two of the later studies reused imputed data that carried the same leak, and the third used proxies for the outcome as inputs. With the errors corrected, the complex models did not perform substantively better than logistic regression. Kapoor and Narayanan's survey, published in Patterns in 2023, found leakage in 294 papers across 17 scientific fields.