Null-hacking, a lurking problem

Pre-registration of analysis plans involves making data-analysis decisions before the data is run in order to prevent flexibly re-running it until a specific result appears (p-hacking). Just because a model and result is pre-registered, however, does not make it reflective of underlying reality. The complement to p-hacking, null-hacking, is the use of the same questionable research practices to re-analyze open data to return a null finding. We provide a vocabulary for null-hacking and introduce the threat it poses. Null-hacking forces consideration of model fit to compare pre-registered and ‘alternative’ models. The reason null-hacking cannot be ignored is a null-hacked model can easily provide better fit to the data than a pre-registered one. Model fit, however, is a precarious problem, focusing just on model fit by only selecting a ‘best fitting model’ eliminates pre-registration, while giving default preference to pre-registered results ignores how well our models can represent the data. We provide a beginning solution aimed at retaining the advantage and justifications of pre-registration, while including model fit, and providing protection against null-hacking. We call this Fully-Informed Model Pre-registration and it involves strict supervised machine learning to maximize local model fit within heavily pre-specified decisions. This solution maximizes local model fit, eliminating the only justifications null-hacked results have. It is not yet a complete solution but merely the groundwork for why other approaches may be insufficient and what a future solution may look like.

1. 1
   RETRACTED ARTICLE: High replicability of newly discovered social-behavioural findings is achievable

   Nature Human Behaviour202327 citationsDataRank 0.500