How to Read Meta-Analyses of Efficacy Studies and Not Go Astray. A Primer for Psychotherapy Practitioners

Afonso, J., Ramirez-Campillo, R., Clemente, F. M., Büttner, F. C., & Andrade, R. (2024). The perils of misinterpreting and misusing “publication bias” in meta-analyses: An education review on funnel plot-based methods. Sports Medicine, 54(2), 257–269. https://doi.org/10.1007/s40279-023-01927-9   Google Scholar

American Psychiatric Association (2020). Development process for practice guidelines of the American Psychiatric Association – revised. https://www.psychiatry.org/ getmedia/0b96df17-66a7-4f49-8159-d6522615f047/APA-Guideline-Development-Process.pdf   Google Scholar

American Psychological Association (n.d.). APA clinical practice guideline development. www.apa.org/about/offices/directorates/guidelines/clinical-practice.   Google Scholar

Andersson, E., Aspvall, K., Schettini, G., Kraepelien, M., Särnholm, J., Wergeland, G. J., & Öst, L. G. (2025). Efficacy of metacognitive interventions for psychiatric disorders: A systematic review and meta-analysis. Cognitive Behaviour Therapy, 54(2), 276–302. https://doi.org/10.1080/16506073.2024.2434920   Google Scholar

Andrade, C. (2015). Understanding relative risk, odds ratio, and related terms: As simple as it can get. The Journal of Clinical Psychiatry, 76(7), 857–861. https://doi.org/10.4088/jcp.15f10150   Google Scholar

Andrade, C. (2020). Understanding the basics of meta-analysis and how to read a forest plot: As simple as it gets. The Journal of Clinical Psychiatry, 81(5), Article 20f13698. https://doi.org/10.4088/JCP.20f13698   Google Scholar

APA Presidential Task Force on Evidence-Based Practice. (2006). Evidence-based practice in psychology. The American Psychologist, 61(4), 271–285. https://doi.org/10.1037/0003-066x.61.4.271   Google Scholar

Aromataris, E., Fernandez, R., Godfrey, C. M., Holly, C., Khalil, H., & Tungpunkom, P. (2015). Summarizing systematic reviews: Methodological development, conduct and reporting of an umbrella review approach. JBI Evidence Implementation, 13(3), 132–140. https://doi.org/10.1097/xeb.0000000000000055   Google Scholar

Baker, S. G., & Kramer, B. S. (2002). The transitive fallacy for randomized trials: If A bests B and B bests C in separate trials, is A better than C? BMC Medical Research Methodology, 2, Article 13. https://doi.org/10.1186/1471-2288-2-13   Google Scholar

Baker, T. B., McFall, R. M., & Shoham, V. (2008). Current status and future prospects of clinical psychology: Toward a scientifically principled approach to mental and behavioral health care. Psychological Science in the Public Interest, 9(2), 67–103. https://doi.org/10.1111/j.1539-6053.2009.01036.x   Google Scholar

Baker, W. L., Michael White, C., Cappelleri, J. C., Kluger, J., Coleman, C. I., From the Health Outcomes, Policy, and Economics (HOPE) Collaborative Group. (2009). Understanding heterogeneity in meta‐analysis: The role of meta‐regression. International Journal of Clinical Practice, 63(10), 1426–1434. https://doi.org/10.1111/j.1742-1241.2009.02168.x   Google Scholar

Begg, C. B., & Mazumdar, M. (1994). Operating characteristics of a rank correlation test for publication bias. Biometrics, 50(4), 1088–1101.   Google Scholar

Blease, C., Kelley, J. M., & Trachsel, M. (2018). Informed consent in psychotherapy: Implications of evidence-based practice. Journal of Contemporary Psychotherapy, 48, 69–78. https://doi.org/10.1007/s10879-017-9372-9   Google Scholar

Boot, W. R., Simons, D. J., Stothart, C., & Stutts, C. (2013). The pervasive problem with placebos in psychology: Why active control groups are not sufficient to rule out placebo effects. Perspectives on Psychological Science, 8(4), 445–454. https://doi.org/10.1177/1745691613491271   Google Scholar

Borenstein, M., Higgins, J. P., Hedges, L. V., & Rothstein, H. R. (2017). Basics of meta‐analysis: I2 is not an absolute measure of heterogeneity. Research Synthesis Methods, 8(1), 5–18. https://doi.org/10.1002/jrsm.1230   Google Scholar

Boutron, I., Page, M. J., Higgins, J. P. T., Altman, D. G., Lundh, A., & Hróbjartsson, A. (2023). Chapter 7: Considering bias and conflicts of interest among the included studies. In Higgins, J. P. T., Thomas, J., Chandler, J., Cumpston, M., Li, T., Page, M. J., Welch, V. A. (Eds.), Cochrane Handbook for Systematic Reviews of Interventions version 6.4. Cochrane. https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-07   Google Scholar

Bowes, S. M., Ammirati, R. J., Costello, T. H., Basterfield, C., & Lilienfeld, S. O. (2020). Cognitive biases, heuristics, and logical fallacies in clinical practice: A brief field guide for practicing clinicians and supervisors. Professional Psychology: Research and Practice, 51(5), 435–445. https://doi.org/10.1037/pro0000309   Google Scholar

Buchman, D., & Rosenbaum, D. (2024). Psychedelics in PeRil: The commercial determinants of health, financial entanglements and population health ethics. Public Health Ethics, 17(1–2), 24–39. https://doi.org/10.1093/phe/phae002   Google Scholar

Burns, P. B., Rohrich, R. J., & Chung, K. C. (2011). The levels of evidence and their role in evidence-based medicine. Plastic and Reconstructive Surgery, 128(1), 305–310. https://doi.org/10.1097/prs.0b013e318219c171   Google Scholar

Caldwell, D. M., Ades, A. E., & Higgins, J. P. T. (2005). Simultaneous comparison of multiple treatments: Combining direct and indirect evidence. BMJ, 331(7521), 897–900. https://doi.org/10.1136/bmj.331.7521.897   Google Scholar

Chan, A. W., Hróbjartsson, A., Haahr, M. T., Gøtzsche, P. C., & Altman, D. G. (2004). Empirical evidence for selective reporting of outcomes in randomized trials: Comparison of protocols to published articles. JAMA, 291(20), 2457–2465. https://doi.org/10.1001/jama.291.20.2457   Google Scholar

Ciharova, M., Karyotaki, E., Miguel, C., Walsh, E., de Ponti, N., Amarnath, A., van Ballegooijen, W., Riper, H., Arroll, B., & Cuijpers, P. (2024). Amount and frequency of psychotherapy as predictors of treatment outcome for adult depression: A meta-regression analysis. Journal of Affective Disorders, 359, 92–99. https://doi.org/10.1016/j.jad.2024.05.070   Google Scholar

Cohn, L. D., & Becker, B. J. (2003). How meta-analysis increases statistical power. Psychological Methods, 8(3), 243–253. https://doi.org/10.1037/1082-989x.8.3.243   Google Scholar

Copay, A. G., Subach, B. R., Glassman, S. D., Polly Jr., D. W., & Schuler, T. C. (2007). Understanding the minimum clinically important difference: A review of concepts and methods. The Spine Journal, 7(5), 541–546. https://doi.org/10.1016/j.spinee.2007.01.008   Google Scholar

Cuijpers, P., & Cristea, I. A. (2016). How to prove that your therapy is effective, even when it is not: A guideline. Epidemiology and Psychiatric Sciences, 25(5), 428–435. https://doi.org/10.1017/s2045796015000864   Google Scholar

Cuijpers, P., Harrer, M., Miguel, C., Ciharova, M., Papola, D., Basic, D., Botella, C., Cristea, I., de Ponti, N., Donker, T., Driessen, E., Franco, P., Gómez-Gómez, I., Hamblen, J., Jiménez-Orenga, N., Karyotaki, E., Keshen, A., Linardon, J., Motrico, E., Matbouriahi, M., Panagiotopoulou, O. M., Pfund, R. A., Plessen, C. Y., Riper, H., Schnurr, P. P., Sijbrandij, M., Toffolo, M. B. J., Tong, L., van Ballegooijen, W., van der Ven, E., van Straten, A., Wang, Y., & Furukawa, T. A. (2025). Cognitive behavior therapy for mental disorders in adults: A unified series of meta-analyses. JAMA Psychiatry, 82(6), 563–571. https://doi.org/10.1001/jamapsychiatry.2025.0482   Google Scholar

Cuijpers, P., Karyotaki, E., de Wit, L., & Ebert, D. D. (2020). The effects of fifteen evidence-supported therapies for adult depression: A meta-analytic review. Psychotherapy Research, 30(3), 279–293. https://doi.org/10.1080/10503307.2019.1649732   Google Scholar

Cumming, G., & Finch, S. (2005). Inference by eye: Confidence intervals and how to read pictures of data. American Psychologist, 60(2), 170–180. https://doi.org/10.1037/0003-066x.60.2.170   Google Scholar

Davies, K. S. (2011). Formulating the evidence based practice question: A review of the frameworks. Evidence Based Library and Information Practice, 6(2), 75–80. https://doi.org/10.18438/B8WS5N   Google Scholar

Davis, A. K., Barrett, F. S., May, D. G., Cosimano, M. P., Sepeda, N. D., Johnson, M. W., Finan, P. H., & Griffiths, R. R. (2021). Effects of psilocybin-assisted therapy on major depressive disorder: A randomized clinical trial. JAMA Psychiatry, 78(5), 481–489. https://doi.org/10.1001/jamapsychiatry.2020.3285   Google Scholar

Deeks, J. J., Higgins, J. P. T., & Altman, D. G. (2023). Chapter 10: Analysing data and undertaking meta-analyses. In J. P. T. Higgins, J. Thomas, J. Chandler, M. Cumpston, T. Li, M. J. Page, V. A. Welch (Eds.), Cochrane Handbook for Systematic Reviews of Interventions version 6.4. Cochrane. https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-10   Google Scholar

Deleuran, D. H. K., Skov, O., & Bo, S. (2024). Prolonged exposure for posttraumatic stress disorder in patients exhibiting psychotic symptoms: A scoping review. Clinical Psychology & Psychotherapy, 31(4), Article e3027. https://doi.org/10.1002/cpp.3027   Google Scholar

Dozois, D. J. A., Mikail, S. F., Alden, L. E., Bieling, P. J., Bourgon, G., Clark, D. A., Drapeau, M., Gallson, D., Greenberg, L., Hunsley, J., & Johnston, C. (2014). The CPA Presidential Task Force on Evidence-Based Practice of Psychological Treatments. Canadian Psychology/Psychologie canadienne, 55(3), 153–160. https://doi.org/10.1037/a0035767   Google Scholar

Duval, S., & Tweedie, R. (2000). Trim and fill: A simple funnel-plot–based method of testing and adjusting for publication bias in meta-analysis. Biometrics, 56(2), 455–463. https://doi.org/10.1111/j.0006-341x.2000.00455.x   Google Scholar

Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. BMJ, 315(7109), 629–634. https://doi.org/10.1136/bmj.315.7109.629   Google Scholar

Evans, D. (2003). Hierarchy of evidence: A framework for ranking evidence evaluating healthcare interventions. Journal of Clinical Nursing, 12(1), 77–84. https://doi.org/10.1046/j.1365-2702.2003.00662.x   Google Scholar

Furukawa, T. A. (1999). From effect size into number needed to treat. The Lancet, 353(9165), Article 1680. https://doi.org/10.1016/s0140-6736(99)01163-0   Google Scholar

Furukawa, T. A., & Leucht, S. (2011). How to obtain NNT from Cohen’s d: Comparison of two methods. PloS One, 6(4), Article e19070. https://doi.org/10.1371/journal.pone.0019070   Google Scholar

Glass, G. V. (1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5(10), 3–8. https://doi.org/10.3102/0013189X005010003   Google Scholar

Grubaugh, A. L., Veronee, K., Ellis, C., Brown, W., & Knapp, R. G. (2017). Feasibility and efficacy of prolonged exposure for PTSD among individuals with a psychotic spectrum disorder. Frontiers in Psychology, 8, Article 977. https://doi.org/10.3389/fpsyg.2017.00977   Google Scholar

Gyani, A., Shafran, R., Layard, R., & Clark, D. M. (2013). Enhancing recovery rates: Lessons from year one of IAPT. Behaviour Research and Therapy, 51(9), 597–606. https://doi.org/10.1016/j.brat.2013.06.004   Google Scholar

Gyani, A., Shafran, R., Myles, P., & Rose, S. (2014). The gap between science and practice: How therapists make their clinical decisions. Behavior Therapy, 45(2), 199–211. https://doi.org/10.1016/j.beth.2013.10.004   Google Scholar

Hannan, C., Lambert, M. J., Harmon, C., Nielsen, S. L., Smart, D. W., Shimokawa, K., & Sutton, S. W. (2005). A lab test and algorithms for identifying clients at risk for treatment failure. Journal of Clinical Psychology, 61(2), 155–163. https://doi.org/10.1002/jclp.20108   Google Scholar

Harrer, M., Cuijpers, P., Schuurmans, L. K., Kaiser, T., Buntrock, C., van Straten, A., & Ebert, D. (2023). Evaluation of randomized controlled trials: A primer and tutorial for mental health researchers. Trials, 24(1), Article 562. https://doi.org/10.1186/s13063-023-07596-3   Google Scholar

Hedges, L. V. (2025). Effect sizes for experimental research. British Journal of Mathematical and Statistical Psychology. https://doi.org/10.1111/bmsp.12389   Google Scholar

Higgins, J. P., & Thompson, S. G. (2002). Quantifying heterogeneity in a meta-analysis. Statistics in Medicine, 21(11), 1539–1558. https://doi.org/10.1002/sim.1186   Google Scholar

Higgins, J. P., Thompson, S. G., Deeks, J. J., & Altman, D. G. (2003). Measuring inconsistency in meta-analyses. BMJ, 327(7414), 557–560. https://doi.org/10.1136/bmj.327.7414.557   Google Scholar

Higgins, J. P. T., Savović, J., Page, M. J., Elbers, R. G., & Sterne, J. A. C. (n.d.). Chapter 8: Assessing risk of bias in a randomized trial. In J. P. T. Higgins, J. Thomas, J. Chandler, M. Cumpston, T. Li, M. J. Page, V. A. Welch (Eds.), Cochrane Handbook for Systematic Reviews of Interventions version 6.4. Cochrane. https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-08   Google Scholar

Higgins, J. P. T., Thomas, J., Chandler, J., Cumpston, M., Li, T., Page, M. J., & Welch, V. A. (Eds.). (2024). Cochrane Handbook for Systematic Reviews of Interventions version 6.5. www.training.cochrane.org/handbook.   Google Scholar

Hutton, B., Salanti, G., Caldwell, D. M., Chaimani, A., Schmid, C. H., Cameron, C., Ioannidis, J. P., Straus, S., Thorlund, K., Jansen, J. P., Mulrow, C., Catalá-López, F., Gøtzsche, P. C., Dickersin, K., Boutron, I., Altman, D. G., & Moher, D. (2015). The PRISMA extension statement for reporting of systematic reviews incorporating network meta-analyses of health care interventions: Checklist and explanations. Annals of Internal Medicine, 162(11), 777–784. https://doi.org/10.7326/m14-2385   Google Scholar

IntHout, J., Ioannidis, J. P. A., Rovers, M. M., & Goeman, J. J. (2016). Plea for routinely presenting prediction intervals in meta-analysis. BMJ Open, 6, Article e010247. https://doi.org/10.1136/bmjopen-2015-010247   Google Scholar

Ioannidis, J. P. (2005). Why most published research findings are false. PLoS Medicine, 2(8), Article e124. https://doi.org/10.1371/journal.pmed.0020124   Google Scholar

Jacobson, N. S., & Truax, P. (1991). Clinical significance: A statistical approach to denning meaningful change in psychotherapy research. Journal of Consulting and Clinical Psychologv, 59(1), 12–19. https://doi.org/10.1037//0022-006x.59.1.12   Google Scholar

JASP Team (2024). JASP (Version 0.18.3) [computer software].   Google Scholar

Jobst, A., Brakemeier, E. -L., Buchheim, A., Caspar, F., Cuijpers, P., Ebmeier, K. P., Falkai, P., Jan van der Gaag, R., Gaebel, W., Herpertz, S., Kurimay, T., Sabaß, L., Schnell, K., Schramm, E., Torrent, C., Wasserman, D., Wiersma, J., & Padberg, F. (2016). European Psychiatric Association Guidance on psychotherapy in chronic depression across Europe. European Psychiatry, 33(1), 18–36. https://doi.org/10.1016/j.eurpsy.2015.12.003   Google Scholar

Kirsch, I. (2005). Placebo psychotherapy: Synonym or oxymoron? Journal of Clinical Psychology, 61(7), 791–803. https://doi.org/10.1002/jclp.20126   Google Scholar

Kowalski, J. (2024). Possible errors in a meta-analysis on the efficacy of psychodynamic therapy in social anxiety disorder (Qiqi Zhang et al., 2022). Psychiatry Research, 342, Article 116174. https://doi.org/10.1016/j.psychres.2024.116174   Google Scholar

Kowalski, J., Blaut, A., Dragan, M., Farley, D., Pankowski, D., Sanna, K., Śliwerski, A., & Wiśniowska, J. (2024). Systematyczny narracyjny przegląd metaanaliz badań nad skutecznością psychoterapii poznawczo-behawioralnej i zaleceń terapeutycznych opublikowanych między 2010 a 2023. [Systematic narrative review of meta-analyses of cognitive-behavioural psychotherapies clinical trials and treatment guidelines published between 2010 and 2023]. Polskie Towarzystwo Terapii Poznawczej i Behawioralnej.   Google Scholar

Kowalski, J., Elżanowski, A., & Śliwerski, A. (2023). A review of selected psychotherapies for PTSD, their efficacy and treatment guidelines in adults. Psychiatria Polska, 58(2), 315–328. https://doi.org/10.12740/pp/onlinefirst/157105   Google Scholar

Kraus, D. R., Castonguay, L., Boswell, J. F., Nordberg, S. S., Hayes, J. A. (2011). Therapist effectiveness: Implications for accountability and patient care. Psychotherapy Research, 21(3), 267–276. https://doi.org/10.1080/10503307.2011.563249   Google Scholar

Kvarven, A., Strømland, E., & Johannesson, M. (2020). Comparing meta-analyses and preregistered multiple-laboratory replication projects. Nature Human Behaviour, 4(4), 423–434. https://doi.org/10.1038/s41562-019-0787-z   Google Scholar

Lakens, D. (2013). Calculating and reporting effect sizes to facilitate cumulative science: A practical primer for t-tests and ANOVAs. Frontiers in Psychology, 4, Article 863. https://doi.org/10.3389/fpsyg.2013.00863   Google Scholar

Lau, J., Ioannidis, J. P., Terrin, N., Schmid, C. H., Olkin, I. (2006). The case of the misleading funnel plot. BMJ, 333(7568), 597–600. https://doi.org/10.1136/bmj.333.7568.597   Google Scholar

Lemarchand, C., Chopin, R., Paul, M., Braillon, A., Cosgrove, L., Cristea, I., Fried, E. I., Turner, E. H., & Naudet, F. (2024). Fragile promise of psychedelics in psychiatry. BMJ, 387, Article e080391. https://doi.org/10.1136/bmj-2024-080391   Google Scholar

Leon, A. C. (2011). Comparative effectiveness clinical trials in psychiatry: Superiority, noninferiority, and the role of active comparators. Journal of Clinical Psychiatry, 72(10), 1344–1349. https://doi.org/10.4088/jcp.10m06089whi   Google Scholar

Levi, O., Bar‐Haim, Y., Kreiss, Y., & Fruchter, E. (2016). Cognitive–behavioural therapy and psychodynamic psychotherapy in the treatment of combat‐related post‐traumatic stress disorder: A comparative effectiveness study. Clinical Psychology & Psychotherapy, 23(4), 298–307. https://doi.org/10.1002/cpp.1969   Google Scholar

Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009). The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: Explanation and elaboration. Journal of Clinical Epidemiology, 62(10), 1–34. https://doi.org/10.1016/j.jclinepi.2009.06.006   Google Scholar

Lilienfeld, S. O., Ritschel, L. A., Lynn, S. J., Cautin, R. L., & Latzman, R. D. (2014). Why ineffective psychotherapies appear to work: A taxonomy of causes of spurious therapeutic effectiveness. Perspectives on Psychological Science, 9(4), 355–387. https://doi.org/10.1177/1745691614535216   Google Scholar

Magnusson, K. (2025). Interpreting Cohen’s d effect size: An interactive visualization (Version 2.6.0). R Psychologist. https://rpsychologist.com/cohend   Google Scholar

Mallard Swanson, K., Song, J., Beristianos, M., Aajmain, S., Lane, J. E. M., Landy, M. S. H., Suvak, M. K., Shields, N., Monson, C. M., & Stirman, S. W. (2021). A glimpse into the “black box”: Which elements of consultation in an EBP are associated with client symptom change and therapist fidelity? Implementation Research and Practice, 2. https://doi.org/10.1177/26334895211051791   Google Scholar

McAleavey, A. A. (2024). When (not) to rely on the reliable change index: A critical appraisal and alternatives to consider in clinical psychology. Clinical Psychology: Science and Practice, 31(3), 351–366. https://doi.org/10.1037/cps0000203   Google Scholar

Methley, A. M., Campbell, S., Chew-Graham, C., McNally, R., & Cheraghi-Sohi, S. (2014). PICO, PICOS and SPIDER: A comparison study of specificity and sensitivity in three search tools for qualitative systematic reviews. BMC Health Services Research, 14(1), Article 579. https://doi.org/10.1186/s12913-014-0579-0   Google Scholar

Michopoulos, I., Furukawa, T. A., Noma, H., Kishimoto, S., Onishi, A., Ostinelli, E. G., Ciharova, M., Miguel, C., Karyotaki, E., & Cuijpers, P. (2021). Different control conditions can produce different effect estimates in psychotherapy trials for depression. Journal of Clinical Epidemiology, 132, 59–70. https://doi.org/10.1016/j.jclinepi.2020.12.012   Google Scholar

Miguel, C., Harrer, M., Karyotaki, E., Plessen, C. Y., Ciharova, M., Furukawa, T. A., Cristea, I. A., & Cuijpers, P. (2025). Self-reports vs clinician ratings of efficacies of psychotherapies for depression: A meta-analysis of randomized trials. Epidemiology and Psychiatric Sciences, 34, Article e15. https://doi.org/10.1017/s2045796025000095   Google Scholar

Moritz, S., Nestoriuc, Y., Rief, W., Klein, J. P., Jelinek, L., & Peth, J. (2019). It can’t hurt, right? Adverse effects of psychotherapy in patients with depression. European Archives of Psychiatry and Clinical Neuroscience, 269(5), 577–586. https://doi.org/10.1007/s00406-018-0931-1   Google Scholar

Munder, T., Brütsch, O., Leonhart, R., Gerger, H., & Barth, J. (2013). Researcher allegiance in psychotherapy outcome research: An overview of reviews. Clinical Psychology Review, 33(4), 501–511. https://doi.org/10.1016/j.cpr.2013.02.002   Google Scholar

National Institute for Health and Care Excellence (2014). Developing NICE guidelines: The manual. https://www.nice.org.uk/process/pmg20/chapter/introduction   Google Scholar

Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences, 115(11), 2600–2606. https://doi.org/10.1073/pnas.1708274114   Google Scholar

Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., Fidler, F., Hilgard, J., Kline Struhl, M., Nuijten, M. B., Rohrer, J. M., Romero, F., Scheel, A. M., Scherer, L. D., Schönbrodt, F. D., & Vazire, S. (2022). Replicability, robustness, and reproducibility in psychological science. Annual Review of Psychology, 73(1), 719–748. https://doi.org/10.1146/annurev-psych-020821-114157   Google Scholar

Nye, A., Delgadillo, J., & Barkham, M. (2023). Efficacy of personalized psychological interventions: A systematic review and meta-analysis. Journal of Consulting and Clinical Psychology, 91(7), 389–397. https://doi.org/10.1037/ccp0000820   Google Scholar

Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), Article aac4716. https://doi.org/10.1126/science.aac4716   Google Scholar

Page, M. J., Sterne, J. A. C., Boutron, I., Hróbjartsson, A., Kirkham, J. J., Li, T., Lundh, A., Mayo-Wilson, E., McKenzie, J. E., Stewart, L. A., Sutton, A. J., Bero, L., Dunn, A. G., Dwan, K., Elbers, R. G., Kanukula, R., Meerpohl, J. J., Turner, E. H., & Higgins, J. P. T. (2023). ROB-ME: A tool for assessing risk of bias due to missing evidence in systematic reviews with meta-analysis. BMJ, 383, Article e076754. https://doi.org/10.1136/bmj-2023-076754   Google Scholar

Philips, B., & Falkenström, F. (2021). What research evidence is valid for psychotherapy research? Frontiers in Psychiatry, 11, Article 625380. https://doi.org/10.3389/fpsyt.2020.625380   Google Scholar

Riehm, K. E., Azar, M., & Thombs, B. D. (2015). Transparency of outcome reporting and trial registration of randomized controlled trials in top psychosomatic and behavioral health journals: A 5-year follow-up. Journal of Psychosomatic Research, 79(1), 1–12. https://doi.org/10.1016/j.jpsychores.2015.04.010   Google Scholar

Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological bulletin, 86(3), 638–641. https://doi.org/10.1037/0033-2909.86.3.638   Google Scholar

Rothstein, H. R., Sutton, A. J., & Borenstein, M. (Eds.). (2005). Publication bias in meta‐analysis: Prevention, assessment and adjustments. John Wiley & Sons.   Google Scholar

Sandoval-Lentisco, A., Tortajada, M., López-Nicolás, R., López-López, J. A., Wagenmakers, E. J., Sánchez-Meca, J., & Hardwicke, T. E. (2025). Preregistration of psychology meta-analyses: A cross-sectional study of prevalence and practice. Advances in Methods and Practices in Psychological Science, 8(1), Article 25152459241300113. https://doi.org/10.1177/25152459241300113   Google Scholar

Salanti, G. (2012). Indirect and mixed‐treatment comparison, network, or multiple‐treatments meta‐analysis: Many names, many benefits, many concerns for the next generation evidence synthesis tool. Research Synthesis Methods, 3(2), 80–97. https://doi.org/10.1002/jrsm.1037   Google Scholar

Schmidt, C. O., & Kohlmann, T. (2008). When to use the odds ratio or the relative risk? International Journal of Public Health, 53(3), 165–167. https://doi.org/10.1007/s00038-008-7068-3   Google Scholar

Schünemann, H. J., Higgins, J. P. T., Vist, G. E., Glasziou, P., Akl, E. A., Skoetz, N., & Guyatt, G. H. (2024). Chapter 14: Completing ‘Summary of findings’ tables and grading the certainty of the evidence. In J. P. T. Higgins, J. Thomas, J. Chandler, M. Cumpston, T. Li, M. J. Page, V. A. Welch (Eds.), Cochrane Handbook for Systematic Reviews of Interventions version 6.5.1. Cochrane. https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-14   Google Scholar

Sedgwick, P. (2015a). How to read a forest plot in a meta-analysis. BMJ, 351, Article h4028 https://doi.org/10.1136/bmj.h4028   Google Scholar

Sedgwick, P. (2015b). What is publication bias in a meta-analysis? BMJ, 351, Article h4419. https://doi.org/10.1136/bmj.h4419   Google Scholar

Sedgwick, P., & Marston, L. (2015). How to read a funnel plot in a meta-analysis. BMJ, 351, Article h4718. https://doi.org/10.1136/bmj.h4718   Google Scholar

Seegan, P. L., Miller, L., Young, A. S., Parrish, C., Cullen, B., & Reynolds, E. K. (2023). Enhancing quality of care through evidence-based practice: Training and supervision experiences. American Journal of Psychotherapy, 76(3), 100–106. https://doi.org/10.1176/appi.psychotherapy.20220015   Google Scholar

Shadish, W. R., & Lecy, J. D. (2015). The meta‐analytic big bang. Research Synthesis Methods, 6(3), 246–264. https://doi.org/10.1002/jrsm.1132   Google Scholar

Shea, B. J., Grimshaw, J. M., Wells, G. A., Boers, M., Andersson, N., Hamel, C., Porter, A. C., Tugwell, P., Moher, D., & Bouter, L. M. (2007). Development of AMSTAR: A measurement tool to assess the methodological quality of systematic reviews. BMC Medical Research Methodology, 7, Article 10. https://doi.org/10.1186/1471-2288-7-10   Google Scholar

Shea, B. J., Bouter, L. M., Peterson, J., Boers, M., Andersson, N., Ortiz, Z., Ramsay, T., Bai, A., Shukla, V. K., & Grimshaw, J. M. (2007). External validation of a measurement tool to assess systematic reviews (AMSTAR). PloS One, 2(12), Article e1350. https://doi.org/10.1371/journal.pone.0001350   Google Scholar

Shea, B. J., Hamel, C., Wells, G. A., Bouter, L. M., Kristjansson, E., Grimshaw, J., Henry, D. A., & Boers, M. (2009). AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. Journal of Clinical Epidemiology, 62(10), 1013–1020. https://doi.org/10.1016/j.jclinepi.2008.10.009   Google Scholar

Shea, B. J., Reeves, B. C., Wells, G., Thuku, M., Hamel, C., Moran, J., Moher, D., Tugwell, P., Welch, V., Kristjansson, E., & Henry, D. A. (2017). AMSTAR 2: A critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ, 358, Article j4008. https://doi.org/10.1136/bmj.j4008   Google Scholar

Simmonds, M. (2015). Quantifying the risk of error when interpreting funnel plots. Systematic Reviews, 4, Article 24. https://doi.org/10.1186/s13643-015-0004-8   Google Scholar

Sleight, P. (2000). Debate: Subgroup analyses in clinical trials: Fun to look at-but don’t believe them! Trials, 1(1), 25–27. https://doi.org/10.1186/cvm-1-1-025   Google Scholar

Smith, M. L., & Glass, G. V. (1977). Meta-analysis of psychotherapy outcome studies. American Psychologist, 32(9), 752–760. https://doi.org/10.1037/0003-066X.32.9.752   Google Scholar

Spring, B. (2007). Evidence‐based practice in clinical psychology: What it is, why it matters; what you need to know. Journal of Clinical Psychology, 63(7), 611–631. https://doi.org/10.1002/jclp.20373   Google Scholar

Sterne, J. A. C., & Egger, M. (2001). Funnel plots for detecting bias in meta-analysis: Guidelines on choice of axis. Journal of Clinical Epidemiology, 54(10), 1046–1055. https://doi.org/10.1016/s0895-4356(01)00377-8   Google Scholar

Sterne, J. A. C., Becker, B. J., & Egger, M. (2005). The funnel plot. In H. R. Rothstein, A. J. Sutton, M. Borenstein, M. (Eds.), Publication bias in meta‐analysis: Prevention, assessment and adjustments (pp. 73–98). Wiley. https://doi.org/10.1002/0470870168.ch5   Google Scholar

Sterne, J. A. C., Hernán, M. A., Reeves, B. C., Savović, J., Berkman, N. D., Viswanathan, M., Henry, D., Altman, D. G., Ansari, M. T., Boutron, I., Carpenter, J. R., Chan, A. -W., Churchill, R., Deeks, J. J., Hróbjartsson, A., Kirkham, J., Jüni, P., Loke, Y. K., Pigott, T. D., Ramsay, C. R., Rothstein, H. R., Sandhu, L., Santaguida, P. L., Schünemann, H. J., Shea, B., Shrier, I., Tugwell, P., Turner, L., Valentine, J. C., Waddington, H., Waters, E., Wells, G. A., Whiting, P. F., & Higgins, J. P. T. (2016). ROBINS-I: A tool for assessing risk of bias in non-randomised studies of interventions. BMJ, 355, Article i4919. https://doi.org/10.1136/bmj.i4919   Google Scholar

Sterne, J. A. C., Savović, J., Page, M. J., Elbers, R. G., Blencowe, N. S., Boutron, I., Cates, C. J., Cheng, H. -Y., Corbett, M. S., Eldridge, S. M., Emberson, J. R., Hernán, M. A., Hopewell, S., Hróbjartsson, A., Junqueira, D. R., Jüni, P., Kirkham, J. J., Lasserson, T., Li, T., McAleenan, A., Reeves, B. C., Shepperd, S., Shrier, I., Stewart, L. A., Tilling, K., White, I. R., Whiting, P. F., & Higgins, J. P. T. (2019). RoB 2: A revised tool for assessing risk of bias in randomised trials. BMJ, 366, Article l4898. https://doi.org/10.1136/bmj.l4898   Google Scholar

Sterne, J. A. C., Hernan, M. A., McAleenan, A., Reeves, B. C., & Higgins, J. P. T. (2023). Chapter 25: Assessing risk of bias in a non-randomized study. In J. P. T. Higgins, J. Thomas, J. Chandler, M. Cumpston, T. Li, M. J. Page, V. A. Welch (Eds.), Cochrane Handbook for Systematic Reviews of Interventions version 6.4. Cochrane. www.training.cochrane.org/handbook   Google Scholar

Stoll, M., Mancini, A., Hubenschmid, L., Dreimüller, N., König, J., Cuijpers, P., Barth, J., & Lieb, K. (2020). Discrepancies from registered protocols and spin occurred frequently in randomized psychotherapy trials—a meta-epidemiologic study. Journal of Clinical Epidemiology, 128, 49–56. https://doi.org/10.1016/j.jclinepi.2020.08.013   Google Scholar

Tolin, D. F., McKay, D., Forman, E. M., Klonsky, E. D., & Thombs, B. D. (2015). Empirically supported treatment: Recommendations for a new model. Clinical Psychology: Science and Practice, 22(4), 317–338. https://doi.org/10.1037/h0101729   Google Scholar

Tolin, D. F., Grasso, D., Boness, C. L., Beck, J. G., Keane, T. M., Leichsenring, F., Olatunji, B. O., Otto, M. W., & Weinand, J. (2025). A proposed definition of psychological treatment and its relation to empirically supported treatments. Clinical Psychology: Science and Practice, 32(3), 213–225. https://doi.org/10.1037/cps0000220   Google Scholar

van Minnen, A., Zoellner, L. A., Harned, M. S., & Mills, K. (2015). Changes in comorbid conditions after prolonged exposure for PTSD: A literature review. Current Psychiatry Reports, 17(3), Article 549. https://doi.org/10.1007/s11920-015-0549-1   Google Scholar

Walfish, S., McAlister, B., O’Donnell, P., & Lambert, M. J. (2012). An investigation of self-assessment bias in mental health providers. Psychological Reports, 110(2), 639–644. https://doi.org/10.2466/02.07.17.pr0.110.2.639-644   Google Scholar

Watts, S. E., Turnell, A., Kladnitski, N., Newby, J. M., & Andrews, G. (2015). Treatment-as-usual (TAU) is anything but usual: A meta-analysis of CBT versus TAU for anxiety and depression. Journal of Affective Disorders, 175, 152–167. https://doi.org/10.1016/j.jad.2014.12.025   Google Scholar

Whiting, P., Savović, J., Higgins, J. P., Caldwell, D. M., Reeves, B. C., Shea, B., Davies, P., Kleijnen, J., Churchill, R., & ROBIS group (2016). ROBIS: A new tool to assess risk of bias in systematic reviews was developed. Journal of Clinical Epidemiology, 69, 225–234. https://doi.org/10.1016/j.jclinepi.2015.06.005   Google Scholar