Adem, Muna
ORCID
Department of Sociology, University of Maryland, College Park 47405, USA
Adriaans, Jule
ORCID
Faculty of Sociology, University of Bielefeld, Bielefeld 33615, Germany
Akdeniz, Esra
ORCID
School of Medicine, Marmara University, Istanbul 34722, Turkey
Alvarez-Benjumea, Amalia
ORCID
The Institute of Public Goods and Policies (IPP), Centro de Ciencias Humanas y Sociales—Consejo Superior de Investigaciones Científicas, Madrid 28003, Spain
Andersen, Henrik K.
ORCID
Institute of Sociology, Chemnitz University of Technology, Chemnitz 09126, Germany
Auer, Daniel
ORCID
Social and Political Science, Collegio Carlo Alberto, Turin 10122, Italy
Azevedo, Flavio
ORCID
Department of Interdisciplinary Social Science, Utrecht University, Utrecht 3584CH, Netherlands
Bahnsen, Oke
ORCID
School of Social Sciences, University of Mannheim, Mannheim 68159, Germany
Balzer, Dave
ORCID
Institute of Sociology, Johannes Gutenberg University Mainz, Mainz 55128, Germany
Bauer, Paul C.
ORCID
Institute of Political Science, University of Freiburg, Freiburg im Breisgau 79085, Germany; Institute of Statistics, LMU Munich, Munich, 79098, Germany
Bol, Thijs
ORCID
Department of Sociology, University of Amsterdam, Amsterdam 1012 WP, Netherlands
Bostic, Amie
ORCID
Department of Sociology, The University of Texas Rio Grande Valley, Edinburg TX 78539, USA
Brzozowska, Zuzanna
ORCID
Vienna Institute of Demography, Austrian Academy of Sciences, Vienna, Austria & Austrian National Public Health Institute (Gesundheit Österreich GmbH, GÖG)
Burger, Kaspar
ORCID
Department of Education, University of Potsdam, Potsdam, 14469, Germany; Jacobs Center for Productive Youth Development, University of Zurich, Zürich, 8006, Switzerland; Social Research Institute, University College London, London WC1H 0AL, UK
Castillo, Juan-Carlos
ORCID
Department of Sociology Universidad de Chile, Millennium Nucleus on Digital Inequalities and Opportunities (NUDOS NCS2022_046), and Center for Social Conflict and Cohesion Studies (COES), University of Chile, Santiago 7800284, Chile
Chan, Nathan
ORCID
Department of Political Science and International Relations, Loyola Marymount University, Los Angeles CA 90045, USA
Christmann, Pablo
ORCID
Data and Research on Society, GESIS—Leibniz-Institute for the Social Sciences, Mannheim 68159, Germany
Connelly, Roxanne
ORCID
School of Social and Political Science, University of Edinburgh, Edinburgh EH8 9LD, Scotland
Eder, Christina
ORCID
Survey Data Curation, GESIS—Leibniz Institute for the Social Sciences, Mannheim 50667, Germany
Eger, Maureen A.
ORCID
Department of Sociology, Umeå University, Umeå 901 87, Sweden; Center for Advanced Study in the Behavioral Sciences, Stanford, CA 94305, USA
Ellerbrock, Simon
ORCID
Mannheim Centre for European Social Research (MZES), University of Mannheim, Mannheim 68131, Germany
Kołczyńska, Marta
ORCID
Department of Research on Social and Institutional Transformations, Institute of Political Studies of the Polish Academy of Sciences, Warsaw 00-625, Poland
Kuk, John
ORCID
Department of Political Science, Michigan State University, East Lansing 48823, USA
Kunißen, Katharina
ORCID
Institute of Sociology, Johannes Gutenberg University Mainz, Mainz 55128, Germany
McManus, Patricia
ORCID
Department of Sociology, Indiana University Bloomington, Bloomington, IN 47405, USA
McWagner, Kyle
ORCID
Department of Political Science, University of Wisconsin-Milwaukee, Milwaukee, WI 53211, USA
Meeusen, Cecil
ORCID
Department of Sociology, Center for Sociological Research, KU Leuven, 3000
Meierrieks, Daniel
ORCID
Research Unit Migration, Integration, Transnationalization, WZB Berlin Social Science Center, Berlin 10785, Germany
Mellon, Jonathan
ORCID
Department of Politics, Westpoint Department of Systems Engineering, M19 2JS
Merhout, Friedolin
ORCID
Department of Sociology and Centre for Social Data Science, University of Copenhagen, 1353
Merk, Samuel
ORCID
Department of School Development, Karlsruhe University of Education, Karlsruhe 76133, Germany
Meyer, Daniel
ORCID
Competence Centre for Regional Development, Federal Institute for Research on Building, Urban Affairs and Spatial Development (BBSR), 03048
Micheli, Leticia
ORCID
Department of Social, Economic & Organisational Psychology, Leiden University, Leiden 2333AK, The Netherlands
Mijs, Jonathan
ORCID
Department of Sociology, Boston University, Boston, MA 02215, USA
Moya, Cristóbal
ORCID
Socio-Economic Panel, Institute for Economic Research, Berlin 10117, Germany
Schachter, Ariela
ORCID
Department of Sociology, Washington University in St. Louis, St. Louis MO 63130, USA
Schaeffer, Merlin
ORCID
Department of Sociology and Centre for Social Data Science, University of Copenhagen, 1353
Schieferdecker, David
ORCID
Institute for Media and Communication Studies, Freie Universität Berlin, Berlin 14195, Germany
Schlueter, Elmar
ORCID
Institute of Sociology, Justus Liebig University Giessen, Giessen 35394, Germany
Schmidt, Katja
ORCID
Department of Social Sciences, Humboldt University Berlin, Berlin, 10117, Germany; Department of Social Sciences, Socio-Economic Panel
Schmidt, Regine
ORCID
Faculty of Social Sciences, Economics, and Business Administration, University of Bamberg, Bamberg 96052, Germany
Teltemann, Janna
ORCID
Institute for Social Sciences, University of Hildesheim, Hildesheim 31141, Germany
Tibajev, Andrey
ORCID
Department of Women's and Children's Health, Uppsala University, Uppsala SE-751 05, Sweden
Tung, Brian
ORCID
Department of Sociology, Washington University in St. Louis, St. Louis 63130, USA
Vagni, Giacomo
ORCID
Social Research Institute (UCL), University College London, London WC1E 6BT, USA
Van Assche, Jasper
ORCID
Center for Social and Cultural Psychology (CESCUP), Université Libre de Bruxelles, Brussels, BE-1050, Belgium; Optentia Research Unit, North-West University, Potchefstroom, 2531, South Africa
van der Linden, Meta
ORCID
Department of Interdisciplinary Social Science, Utrecht University, Utrecht 3584CH, Netherlands
Van Hootegem, Arno
ORCID
Department of Sociology and Human Geography, University of Oslo, Oslo 851, Norway
Vogtenhuber, Stefan
ORCID
Education and Employment, Institute for Advanced Studies, Vienna 1080, Austria
Voicu, Bogdan
ORCID
Research Institute for Quality of Life, Romanian Academy, Bucharest, 010071, Romania; Department of Sociology, Lucian Blaga University of Sibiu, Sibiu, 550024, Romania
Wagemans, Fieke
ORCID
Beleidsvisies, Burgervisies en Gedragingen (Policy Perspectives, Citizen Perspectives, and Behaviors), Netherlands Institute for Social Research, The Hague 2594, Netherlands
Wehl, Nadja
ORCID
Research Cluster ‘The Politics of Inequality’, University of Konstanz, Konstanz 78464, Germany
Werner, Hannah
ORCID
Department of Political Science, University of Zurich, Zürich 8050, Switzerland
Zhang, Nan
ORCID
Faculty of Arts and Science, Kyushu University, Fukuoka 819-0395, Japan
Ziller, Conrad
ORCID
Department of Political Science, University of Duisburg-Essen, Duisburg-Essen 47057, Germany
Zins, Stefan
ORCID
Institute for Employment Research, Federal Employment Agency, Nuremberg 90478, Germany
Żółtak, Tomasz
ORCID
Department of Computational Social Sciences, Institute of Philosophy and Sociology of the Polish Academy of Sciences, Warsaw 00-330, Poland
English
This study investigates researcher variability in computational reproduction, an activity for which it is least expected. Eighty-five independent teams attempted numerical replication of results from an original study of policy preferences and immigration. Reproduction teams were randomly grouped into a ‘transparent group’ receiving original study and code or ‘opaque group’ receiving only a method and results description and no code. The transparent group mostly verified original results (95.7% same sign and p-value cutoff), while the opaque group had less success (89.3%). Seconddecimal place exact numerical reproductions were less common (76.9 and 48.1%). Qualitative investigation of the workflows revealed many causes of error, including mistakes and procedural variations. When curating mistakes, we still find that only the transparent group was reliably successful. Our findings imply a need for transparency, but also more. Institutional checks and less subjective difficulty for researchers ‘doing reproduction’ would help, implying a need for better training. We also urge increased awareness of complexity in the research process and in ‘push button’ replications.
Faculty
Faculté des lettres et des sciences humaines
Department
Département de travail social, politiques sociales et développement global