Bayesian Evidence Synthesis
The integration of multiple studies.
In recent years, the importance of replications has received considerable attention (e.g., Open Science Collaboration, 2015; Baker, 2016; Brandt et al., 2014). However, emphasis has been placed primarily on exact, direct or close replication studies. These studies employ an identical methodology and research design as the initial study, and are thus merely concerned with the statistical reliability of the results. If these results depend on methodological flaws, inferences from all studies will lead to suboptimal or invalid conclusions (Munafò & Smith, 2018). To overcome these limitations, the use of conceptual replications has been advocated (e.g., Munafò & Smith, 2018; Lawlor, Tilling, & Davey Smith, 2017). Specifically, conceptual replications scrutinize the extent to which the initial conclusions hold under different conditions, using varying instruments or operationalizations.
However, established methods such as (Bayesian) meta-analysis and Bayesian updating are not applicable when studies differ conceptually. This is due to the fact that these methods require that the parameter estimates (i) share a common scale, and (ii) result from analyses with identical function forms (Lipsey & Wilson, 2001; Schönbrodt, Wagenmakers, Zehetleitner, & Perugini, 2017; Sutton & Abrams, 2001). Consequently, Kuiper, Buskens, Raub, & Hoijtink (2013) proposed Bayesian Evidence Synthesis (BES), which is built upon the foundation of the Bayes Factor (BF; Kass & Raftery, 1995). This method allows researchers to pool evidence for a specific hypothesis over multiple studies, even if the studies have seemingly incompatible designs.
In the current project, Irene Klugkist and I aim to reveal under which circumstances BES performs inadequately. Additionally, we hope to propose adjustments to the method that improve its performance, so that an increasing number of researchers can benefit from BES.
References
Baker, M. (2016). Reproducibility crisis. Nature, 533(26), 353–366. https://doi.org/10.1038/533452a
Brandt, M. J., IJzerman, H., Dijksterhuis, A., Farach, F. J., Geller, J., Giner-Sorolla, R., … Van’t Veer, A. (2014). The replication recipe: What makes for a convincing replication? Journal of Experimental Social Psychology, 50, 217–224. https://doi.org/10.1016/j.jesp.2013.10.005
Kass, R. E., & Raftery, A. E. (1995). Bayes factors. Journal of the American Statistical Association, 90(430), 773–795.
Kuiper, R. M., Buskens, V., Raub, W., & Hoijtink, H. (2013). Combining statistical evidence from several studies: A method using bayesian updating and an example from research on trust problems in social and economic exchange. Sociological Methods & Research, 42(1), 60–81.
Lawlor, D. A., Tilling, K., & Davey Smith, G. (2017). Triangulation in aetiological epidemiology. International Journal of Epidemiology, dyw314. https://doi.org/10.1093/ije/dyw314
Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-analysis. SAGE publications, Inc.
Munafò, M. R., & Smith, G. D. (2018). Robust research needs many lines of evidence. Nature, 553(7689), 399–401. https://doi.org/10.1038/d41586-018-01023-3
Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251). https://doi.org/10.1126/science.aac4716
Schönbrodt, F. D., Wagenmakers, E.-J., Zehetleitner, M., & Perugini, M. (2017). Sequential hypothesis testing with bayes factors: Efficiently testing mean differences. Psychological Methods, 22(2), 322.
Sutton, A. J., & Abrams, K. R. (2001). Bayesian methods in meta-analysis and evidence synthesis. Statistical Methods in Medical Research, 10(4), 277–303. https://doi.org/10.1177/096228020101000404
