Transparency and reproducibility in artificial intelligence

Benjamin Haibe-Kains; George Alexandru Adam; Ahmed Hosny; Farnoosh Khodakarami; Massive Analysis Quality Control (MAQC) Society Board of Directors; Levi Waldron; Bo Wang; Chris McIntosh; Anna Goldenberg; Anshul Kundaje; Casey S Greene; Tamara Broderick; Michael M Hoffman; Jeffrey T Leek; Keegan Korthauer; Wolfgang Huber; Alvis Brazma; Joelle Pineau; Robert Tibshirani; Trevor Hastie; John P A Ioannidis; John Quackenbush; Hugo J W L Aerts

doi:10.1038/s41586-020-2766-y

Transparency and reproducibility in artificial intelligence

Nature. 2020 Oct;586(7829):E14-E16. doi: 10.1038/s41586-020-2766-y. Epub 2020 Oct 14.

Authors

Benjamin Haibe-Kains^{1

2

3

4

5}, George Alexandru Adam^{6

7}, Ahmed Hosny^{8

9}, Farnoosh Khodakarami^{10

11}; Massive Analysis Quality Control (MAQC) Society Board of Directors; Levi Waldron¹², Bo Wang^{11

6

7

13

14}, Chris McIntosh^{11

7

13}, Anna Goldenberg^{6

7

15

16}, Anshul Kundaje^{17

18}, Casey S Greene^{19

20}, Tamara Broderick²¹, Michael M Hoffman^{10

11

6

7}, Jeffrey T Leek²², Keegan Korthauer^{23

24}, Wolfgang Huber²⁵, Alvis Brazma²⁶, Joelle Pineau^{27

28}, Robert Tibshirani^{29

30}, Trevor Hastie^{29

30}, John P A Ioannidis^{29

30

31

32

33}, John Quackenbush^{34

35

36}, Hugo J W L Aerts^{8

9

37

38}

Collaborators

Massive Analysis Quality Control (MAQC) Society Board of Directors:
Thakkar Shraddha, Rebecca Kusko, Susanna-Assunta Sansone, Weida Tong, Russ D Wolfinger, Christopher E Mason, Wendell Jones, Joaquin Dopazo, Cesare Furlanello

Affiliations

¹ Princess Margaret Cancer Centre, University Health Network, Toronto, Ontario, Canada. bhaibeka@uhnresearch.ca.
² Department of Medical Biophysics, University of Toronto, Toronto, Ontario, Canada. bhaibeka@uhnresearch.ca.
³ Department of Computer Science, University of Toronto, Toronto, Ontario, Canada. bhaibeka@uhnresearch.ca.
⁴ Ontario Institute for Cancer Research, Toronto, Ontario, Canada. bhaibeka@uhnresearch.ca.
⁵ Vector Institute for Artificial Intelligence, Toronto, Ontario, Canada. bhaibeka@uhnresearch.ca.
⁶ Department of Computer Science, University of Toronto, Toronto, Ontario, Canada.
⁷ Vector Institute for Artificial Intelligence, Toronto, Ontario, Canada.
⁸ Artificial Intelligence in Medicine (AIM) Program, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA.
⁹ Radiation Oncology and Radiology, Dana-Farber Cancer Institute, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA.
¹⁰ Princess Margaret Cancer Centre, University Health Network, Toronto, Ontario, Canada.
¹¹ Department of Medical Biophysics, University of Toronto, Toronto, Ontario, Canada.
¹² Department of Epidemiology and Biostatistics and Institute for Implementation Science in Population Health, CUNY Graduate School of Public Health and Health Policy, New York, NY, USA.
¹³ Peter Munk Cardiac Centre, University Health Network, Toronto, Ontario, Canada.
¹⁴ Department of Laboratory Medicine and Pathobiology, University of Toronto, Ontario, Canada.
¹⁵ SickKids Research Institute, Toronto, Ontario, Canada.
¹⁶ Child and Brain Development Program, CIFAR, Toronto, Ontario, Canada.
¹⁷ Department of Genetics, Stanford University School of Medicine, Stanford, CA, USA.
¹⁸ Department of Computer Science, Stanford University, Stanford, CA, USA.
¹⁹ Dept. of Systems Pharmacology and Translational Therapeutics, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA.
²⁰ Childhood Cancer Data Lab, Alex's Lemonade Stand Foundation, Philadelphia, PA, USA.
²¹ Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology, Cambridge, MA, USA.
²² Department of Biostatistics, Johns Hopkins Bloomberg School of Public Health, Baltimore, MD, USA.
²³ Department of Statistics, University of British Columbia, Vancouver, British Columbia, Canada.
²⁴ BC Children's Hospital Research Institute, Vancouver, British Columbia, Canada.
²⁵ European Molecular Biology Laboratory, Genome Biology Unit, Heidelberg, Germany.
²⁶ European Molecular Biology Laboratory, European Bioinformatics Institute, EMBL-EBI, Hinxton, UK.
²⁷ McGill University, Montreal, Quebec, Canada.
²⁸ Montreal Institute for Learning Algorithms, Quebec, Canada.
²⁹ Department of Statistics, Stanford University School of Humanities and Sciences, Stanford, CA, USA.
³⁰ Department of Biomedical Data Science, Stanford University School of Medicine, Stanford, CA, USA.
³¹ Department of Medicine, Stanford University School of Medicine, Stanford, CA, USA.
³² Meta-Research Innovation Center at Stanford (METRICS), Stanford, CA, USA.
³³ Department of Epidemiology and Population Health, Stanford University School of Medicine, Stanford, CA, USA.
³⁴ Department of Biostatistics, Harvard T.H Chan School of Public Health, Boston, MA, USA.
³⁵ Channing Division of Network Medicine, Brigham and Women's Hospital, Boston, MA, USA.
³⁶ Department of Data Science, Dana-Farber Cancer Institute, Boston, MA, USA.
³⁷ Radiology and Nuclear Medicine, Maastricht University, Maastricht, The Netherlands.
³⁸ Cardiovascular Imaging Research Center, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA.

Abstract

Breakthroughs in artificial intelligence (AI) hold enormous potential as it can automate complex tasks and go even beyond human performance. In their study, McKinney et al. showed the high potential of AI for breast cancer screening. However, the lack of methods’ details and algorithm code undermines its scientific value. Here, we identify obstacles hindering transparent and reproducible AI research as faced by McKinney et al., and provide solutions to these obstacles with implications for the broader field.

Transparency and reproducibility in artificial intelligence

Authors

Collaborators

Affiliations

Abstract

Publication types

MeSH terms

Grants and funding