Measuring reproducibility of high-throughput experiments

Reproducibility is essential to reliable scientific discovery in\nhigh-throughput experiments. In this work we propose a unified approach to\nmeasure the reproducibility of findings identified from replicate experiments\nand identify putative discoveries using reproducibility. Unlike the usual\nscalar measures of reproducibility, our approach creates a curve, which\nquantitatively assesses when the findings are no longer consistent across\nreplicates. Our curve is fitted by a copula mixture model, from which we derive\na quantitative reproducibility score, which we call the "irreproducible\ndiscovery rate" (IDR) analogous to the FDR. This score can be computed at each\nset of paired replicate ranks and permits the principled setting of thresholds\nboth for assessing reproducibility and combining replicates. Since our approach\npermits an arbitrary scale for each replicate, it provides useful descriptive\nmeasures in a wide variety of situations to be explored. We study the\nperformance of the algorithm using simulations and give a heuristic analysis of\nits theoretical properties. We demonstrate the effectiveness of our method in a\nChIP-seq experiment.\n