In “The reusable holdout: Preserving validity in adaptive data analysis” published in Science researchers Cynthia Dwork, Vitaly Feldman, Moritz Hardt, Toniann Pitassi, Omer Reingold, Aaron Roth addresses the issue of data analysis adaptivity.
Applying thumb rules such as the same 5% significance test many learn when introduced to scientific method at school sometimes corroborate misleading ‘discoveries’. Data analysis often enough is made through a re-interpretation of statistics. So that conclusions carry much of our models and how we interpret raw data in the first place.
Author Moritz Hardt posted an interesting introduction to the paper in Google Research Blog.