The reliability of replications: a study in computational reproductions
BP2-STS
-
Breznau, Nate
ORCID
Organization and Program Planning, German Institute for Adult Education—Leibniz Center of Lifelong Learning, Bonn 53175, Germany
-
Rinke, Eike Mark
ORCID
School of Politics and International Studies, University of Leeds, Leeds LS2 9JT, UK
-
Wuttke, Alexander
ORCID
Geschwister Scholl Institute, LMU Munich, Munich 81541, Germany
-
Adem, Muna
ORCID
Department of Sociology, University of Maryland, College Park 47405, USA
-
Adriaans, Jule
ORCID
Faculty of Sociology, University of Bielefeld, Bielefeld 33615, Germany
-
Akdeniz, Esra
ORCID
School of Medicine, Marmara University, Istanbul 34722, Turkey
-
Alvarez-Benjumea, Amalia
ORCID
The Institute of Public Goods and Policies (IPP), Centro de Ciencias Humanas y Sociales—Consejo Superior de Investigaciones Científicas, Madrid 28003, Spain
-
Andersen, Henrik K.
ORCID
Institute of Sociology, Chemnitz University of Technology, Chemnitz 09126, Germany
-
Auer, Daniel
ORCID
Social and Political Science, Collegio Carlo Alberto, Turin 10122, Italy
-
Azevedo, Flavio
ORCID
Department of Interdisciplinary Social Science, Utrecht University, Utrecht 3584CH, Netherlands
-
Bahnsen, Oke
ORCID
School of Social Sciences, University of Mannheim, Mannheim 68159, Germany
-
Bai, Ling
void
-
Balzer, Dave
ORCID
Institute of Sociology, Johannes Gutenberg University Mainz, Mainz 55128, Germany
-
Bauer, Paul C.
ORCID
Institute of Political Science, University of Freiburg, Freiburg im Breisgau 79085, Germany; Institute of Statistics, LMU Munich, Munich, 79098, Germany
-
Bauer, Gerrit
ORCID
Department of Sociology, LMU Munich, Munich 80801, Germany
-
Baumann, Markus
ORCID
Office of the Executive Board, GESIS—Leibniz Institute for the Social Sciences, Mannheim 68159, Germany
-
Baute, Sharon
ORCID
Department of Politics and Public Administration, University of Konstanz, Konstanz 78457, Germany
-
Benoit, Verena
ORCID
Faculty of Social Sciences, Economics, and Business Administration, University of Bamberg, Bamberg 96052, Germany
-
Bernauer, Julian
ORCID
Mannheim Centre for European Social Research (MZES), University of Mannheim, Mannheim 68131, Germany
-
Berning, Carl
Institute for Political Science, Johannes Gutenberg University Mainz, Mainz 55099, Germany
-
Berthold, Anna
ORCID
Faculty of Social Sciences, Economics, and Business Administration, University of Bamberg, Bamberg 96052, Germany
-
Bethke, Felix S.
ORCID
Research Department IV: Intrastate Conflict, Peace Research Institute Frankfurt (PRIF), Frankfurt 60329, Germany
-
Biegert, Thomas
ORCID
Department of Social Policy, London School of Economics and Political Science, London WC2A 2AE, UK
-
Blinzler, Katharina
Survey Data Curation, GESIS—Leibniz Institute for the Social Sciences, Mannheim 50667, Germany
-
Blumenberg, Johannes N.
ORCID
Knowledge Exchange and Outreach, GESIS—Leibniz-Institute for the Social Sciences, Mannheim 67067, Germany
-
Bobzien, Licia
ORCID
Faculty of Economics and Social Sciences, University of Potsdam, Potsdam 10117, Germany
-
Bohman, Andrea
ORCID
Department of Sociology, Umeå University, Umeå 901 87, Sweden
-
Bol, Thijs
ORCID
Department of Sociology, University of Amsterdam, Amsterdam 1012 WP, Netherlands
-
Bostic, Amie
ORCID
Department of Sociology, The University of Texas Rio Grande Valley, Edinburg TX 78539, USA
-
Brzozowska, Zuzanna
ORCID
Vienna Institute of Demography, Austrian Academy of Sciences, Vienna, Austria & Austrian National Public Health Institute (Gesundheit Österreich GmbH, GÖG)
-
Burgdorf, Katharina
ORCID
School of Social Sciences, University of Bremen, Bremen 28359, Germany
-
Burger, Kaspar
ORCID
Department of Education, University of Potsdam, Potsdam, 14469, Germany; Jacobs Center for Productive Youth Development, University of Zurich, Zürich, 8006, Switzerland; Social Research Institute, University College London, London WC1H 0AL, UK
-
Busch, Kathrin
ORCID
Independent Researcher
-
Castillo, Juan-Carlos
ORCID
Department of Sociology Universidad de Chile, Millennium Nucleus on Digital Inequalities and Opportunities (NUDOS NCS2022_046), and Center for Social Conflict and Cohesion Studies (COES), University of Chile, Santiago 7800284, Chile
-
Chan, Nathan
ORCID
Department of Political Science and International Relations, Loyola Marymount University, Los Angeles CA 90045, USA
-
Christmann, Pablo
ORCID
Data and Research on Society, GESIS—Leibniz-Institute for the Social Sciences, Mannheim 68159, Germany
-
Connelly, Roxanne
ORCID
School of Social and Political Science, University of Edinburgh, Edinburgh EH8 9LD, Scotland
-
Czymara, Christian S.
ORCID
Institute of Sociology, Goethe-University Frankfurt, Frankfurt 60323, Germany
-
Damian, Elena
ORCID
Epidemiology and Public Health, Sciensano, Brussels 1050, Belgium
-
de Rooij, Eline A.
ORCID
Political Science, Simon Fraser University, Burnaby V5A 1S6, Canada
-
Ecker, Alejandro
ORCID
Institute of Political Science, Heidelberg University, Heidelberg 69115, Germany
-
Edelmann, Achim
ORCID
Médialab Sciences Po, Paris 75007, France
-
Eder, Christina
ORCID
Survey Data Curation, GESIS—Leibniz Institute for the Social Sciences, Mannheim 50667, Germany
-
Eger, Maureen A.
ORCID
Department of Sociology, Umeå University, Umeå 901 87, Sweden; Center for Advanced Study in the Behavioral Sciences, Stanford, CA 94305, USA
-
Ellerbrock, Simon
ORCID
Mannheim Centre for European Social Research (MZES), University of Mannheim, Mannheim 68131, Germany
-
Forke, Anna
void
-
Forster, Andrea
ORCID
Department of Sociology, Utrecht University, Utrecht 3584 CS, Netherlands
-
Freire, Danilo
ORCID
Department of Quantitative Theory and Methods, Emory University, Atlanta 30306, USA
-
Gaasendam, Chris
ORCID
Department of Work and Social Economy, Government of Flanders-Belgium
-
Gavras, Konstantin
ORCID
School of Social Sciences, University of Mannheim, Mannheim 68159, Germany
-
Gayle, Vernon
ORCID
School of Social and Political Science, University of Edinburgh, Edinburgh EH8 9LD, Scotland
-
Gessler, Theresa
ORCID
Kulturwissenschaftliche Fakultät, Europa Universität Viadrina, Frankfurt 15230, Germany
-
Gnambs, Timo
ORCID
Educational Measurement, Leibniz Institute for Educational Trajectories, Bamberg 96047, Germany
-
Godefroidt, Amélie
ORCID
Centre for Research on Peace and Development, KU Leuven, Leuven 3000, Belgium
-
Grömping, Max
ORCID
School of Government and International Relations, Griffith University, Nathan, Queensland 4111, Australia
-
Groß, Martin
ORCID
Department of Sociology, University of Tübingen, Tübingen 72074, Germany
-
Gruber, Stefan
ORCID
Research Data Center and Communication, SHARE BERLIN Institute, Berlin 10115, Germany
-
Gummer, Tobias
ORCID
Data and Research on Society, GESIS—Leibniz Institute for the Social Sciences, Mannheim 68159, Germany
-
Hadjar, Andreas
ORCID
Division Sociology, Social Policy and Social Work, University of Fribourg, Fribourg CH-1700, Switzerland
-
Halbherr, Verena
ORCID
Association for Doctoral Studies Baden-Wuerttemberg 70174, Germany
-
Heisig, Jan Paul
ORCID
Research Group ‘Health and Social Inequality’, WZB Berlin Social Science Center, Berlin 10785, Germany
-
Hellmeier, Sebastian
ORCID
Transformations of Democracy Unit, WZB Berlin Social Science Center, Berlin 10785, Germany
-
Heyne, Stefanie
ORCID
Mannheim Centre for European Social Research (MZES), University of Mannheim, Mannheim 68131, Germany
-
Hirsch, Magdalena
ORCID
Research Unit Migration, Integration, Transnationalization, WZB Berlin Social Science Center, Berlin 10785, Germany
-
Hjerm, Mikael
ORCID
Department of Sociology, Umeå University, Umeå 901 87, Sweden
-
Hochman, Oshrat
ORCID
Data and Research on Society, GESIS—Leibniz Institue for the Social Sciences, Mannheim 68159, Germany
-
Höffler, Jan H.
ORCID
Facultad de Emprendimiento, Negocios y Economía, Universidad Espíritu Santo, ReplicationWiki, and EQ-Lab
-
Hövermann, Andreas
ORCID
Wirtschafts- und Sozialwissenschaftliches Institut (WSI), Hans-Böckler-Foundation, Düsseldorf 40474, Germany
-
Hunger, Sophia
ORCID
SOCIUM—Research Center on Inequality and Social Policy, University of Bremen, Bremen 10785, Germany
-
Hunkler, Christian
ORCID
Berlin Institute for Integration and Migration Research (BIM), Humboldt Universität zu Berlin, Berlin 10099, Germany
-
Huth-Stöckle, Nora
ORCID
School of Human and Social Sciences, University of Wuppertal, Wuppertal 42119, Germany
-
Ignácz, Zsófia S.
ORCID
Institute of Sociology, Goethe-University Frankfurt, Frankfurt 60323, Germany
-
Israel, Sabine
Independent Researcher
-
Jacobs, Laura
ORCID
Department of Political Science, University of Antwerp, Antwerpen 2000, Belgium
-
Jacobsen, Jannes
ORCID
Cluster ‘Data-Methods-Monitoring’, German Center for Integration and Migration Research (DeZIM)
-
Jaeger, Bastian
ORCID
Department of Social Psychology, Tilburg University, Tilburg 5037AB, Netherlands
-
Jungkunz, Sebastian
ORCID
Institute of Political Science and Sociology, University of Bonn, Bonn 53111, Germany
-
Jungmann, Nils
ORCID
Survey Data Curation, GESIS—Leibniz Institute for the Social Sciences, Mannheim 50667, Germany
-
Kanjana, Jennifer
void
-
Kauff, Mathias
ORCID
Department of Psychology, Medical School Hamburg, Hamburg 20457, Germany
-
Khan, Salman
Economics, University of Illinois, Chicago
-
Khatua, Sayak
ORCID
School of Public Policy, Oregon State University, Corvallis 97330, USA
-
Kleinert, Manuel
Institute of Sociology, Justus Liebig University Giessen, Giessen 35394, Germany
-
Klinger, Julia
ORCID
Independent Researcher
-
Kolb, Jan-Philipp
ORCID
Statistisches Bundesamt, Statistisches Bundesamt Wiesbaden, Wiesbaden 67549, Germany
-
Kołczyńska, Marta
ORCID
Department of Research on Social and Institutional Transformations, Institute of Political Studies of the Polish Academy of Sciences, Warsaw 00-625, Poland
-
Kuk, John
ORCID
Department of Political Science, Michigan State University, East Lansing 48823, USA
-
Kunißen, Katharina
ORCID
Institute of Sociology, Johannes Gutenberg University Mainz, Mainz 55128, Germany
-
Kurti Sinatra, Dafina
ORCID
Center for Evaluation, Independent Researcher (Formerly Uni Cologne)
-
Langenkamp, Alexander
ORCID
Institute of Sociology, Goethe-University Frankfurt, Frankfurt 60323, Germany
-
Lee, Robin C.
Department of Sociology, Princeton University
-
Lersch, Philipp M.
ORCID
Socio-Economic Panel, German Institute for Economic Research, Berlin 10117, Germany
-
Liu, David
void
-
Löbel, Lea-Maria
ORCID
Socio-Economic Panel, German Institute for Economic Research, Berlin 10117, Germany
-
Lutscher, Philipp
ORCID
Department of Political Science, University of Oslo, Oslo 0851, Norway
-
Mader, Matthias
ORCID
Department of Philosophy, Politics and Economics, Witen/Herdecke University, Witten 58488, Germany
-
Madia, Joan E.
ORCID
Department of Primary Care and Health Sciences, University of Oxford, Oxford OX11JD, England
-
Malancu, Natalia
ORCID
Swiss Forum for Migration and Population Studies, University of Neuchatel, Neuchâtel 1205, Switzerland
-
Maldonado, Luis
ORCID
Instituto de Sociologia, Pontifical Catholic University of Chile, Santiago 7820436, Chile
-
Marahrens, Helge
ORCID
Massive Data Institute, Georgetown University, Washington D.C. 20057, USA
-
Martin, Nicole
ORCID
Department of Politics, University of Manchester, Manchester M19JS, UK
-
Martinez, Paul
ORCID
Department of Institutional Research, Western Governors University, Millcreek 84107, USA
-
Mayerl, Jochen
ORCID
Institute of Sociology, Chemnitz University of Technology, Chemnitz 09126, Germany
-
Mayorga, Oscar J.
ORCID
Director of Data for Freedom, Equity Research Cooperative, 19107
-
McDonnell, Robert
ORCID
void
-
McManus, Patricia
ORCID
Department of Sociology, Indiana University Bloomington, Bloomington, IN 47405, USA
-
McWagner, Kyle
ORCID
Department of Political Science, University of Wisconsin-Milwaukee, Milwaukee, WI 53211, USA
-
Meeusen, Cecil
ORCID
Department of Sociology, Center for Sociological Research, KU Leuven, 3000
-
Meierrieks, Daniel
ORCID
Research Unit Migration, Integration, Transnationalization, WZB Berlin Social Science Center, Berlin 10785, Germany
-
Mellon, Jonathan
ORCID
Department of Politics, Westpoint Department of Systems Engineering, M19 2JS
-
Merhout, Friedolin
ORCID
Department of Sociology and Centre for Social Data Science, University of Copenhagen, 1353
-
Merk, Samuel
ORCID
Department of School Development, Karlsruhe University of Education, Karlsruhe 76133, Germany
-
Meyer, Daniel
ORCID
Competence Centre for Regional Development, Federal Institute for Research on Building, Urban Affairs and Spatial Development (BBSR), 03048
-
Micheli, Leticia
ORCID
Department of Social, Economic & Organisational Psychology, Leiden University, Leiden 2333AK, The Netherlands
-
Mijs, Jonathan
ORCID
Department of Sociology, Boston University, Boston, MA 02215, USA
-
Moya, Cristóbal
ORCID
Socio-Economic Panel, Institute for Economic Research, Berlin 10117, Germany
-
Neunhoeffer, Marcel
ORCID
School of Social Sciences, LMU Munich
-
Nüst, Daniel
ORCID
Department of Geosciences, Technische Universität Dresden, Dresden 01069, Germany
-
Nygård, Olav
ORCID
Division of Migration, Ethnicity and Society (REMESO), Linköping University, 60174
-
Ochsenfeld, Fabian
Administrative Headquarters, Max Planck Society, Munich 80539, Germany
-
Otte, Gunnar
ORCID
Institute of Sociology, Johannes Gutenberg University Mainz, Mainz 55128, Germany
-
Pechenkina, Anna
ORCID
Department of Political Science, Utah State University, Logan, UT 84321, USA
-
Pickup, Mark
ORCID
Political Science, Simon Fraser University
-
Prosser, Christopher
ORCID
void
-
Raes, Louis
ORCID
Department of Economics, Tilburg University, Tilburg 5037 AB, The Netherlands
-
Ralston, Kevin
ORCID
School of Social and Political Science, University of Edinburgh, Edinburgh EH8 9LD, Scotland
-
Ramos, Miguel
ORCID
Department of Social Policy, Sociology and Criminology, University of Birmingham, Birmingham B15 2TT, UK
-
Reichert, Frank
ORCID
void
-
Roets, Arne
ORCID
Department of Developmental, Personality, and Social Psychology, Ghent University
-
Rogers, Jonathan
ORCID
School of Law, Empirical Research Group, University of California, Los Angeles
-
Ropers, Guido
ORCID
Independent Researcher
-
Samuel, Robin
ORCID
Department of Social Sciences, University of Luxembourg, 4366, Luxembourg
-
Sand, Gregor
ORCID
SHARE Operations, SHARE Berlin Institute, Berlin 10115, Germany
-
Sanhueza Petrarca, Constanza
ORCID
School of Politics and International Relations, Australian National University, Canberra 2132, Australia
-
Schachter, Ariela
ORCID
Department of Sociology, Washington University in St. Louis, St. Louis MO 63130, USA
-
Schaeffer, Merlin
ORCID
Department of Sociology and Centre for Social Data Science, University of Copenhagen, 1353
-
Schieferdecker, David
ORCID
Institute for Media and Communication Studies, Freie Universität Berlin, Berlin 14195, Germany
-
Schlueter, Elmar
ORCID
Institute of Sociology, Justus Liebig University Giessen, Giessen 35394, Germany
-
Schmidt, Katja
ORCID
Department of Social Sciences, Humboldt University Berlin, Berlin, 10117, Germany; Department of Social Sciences, Socio-Economic Panel
-
Schmidt, Regine
ORCID
Faculty of Social Sciences, Economics, and Business Administration, University of Bamberg, Bamberg 96052, Germany
-
Schmidt-Catran, Alexander
ORCID
Institute of Sociology, Goethe-University Frankfurt, Frankfurt 60323, Germany
-
Schmiedeberg, Claudia
ORCID
Department of Sociology, LMU Munich, Munich 80801, Germany
-
Schneider, Jürgen
ORCID
Teacher and Teaching Quality, Leibniz Institute for Research and Information in Education, Frankfurt 60323, Germany
-
Schoonvelde, Martijn
ORCID
Chair group European Politics and Society, University of Groningen, Groningen 9712 EK, Netherlands
-
Schulte-Cloos, Julia
ORCID
Department of Political Science, University of Marburg, Marburg 35037, Germany
-
Schumann, Sandy
ORCID
Department of Security and Crime Science, University College London, London WC1E 6BT, UK
-
Schunck, Reinhard
ORCID
School of Human and Social Sciences, University of Wuppertal, Wuppertal 42119, Germany
-
Seuring, Julian
ORCID
Department Migration, Leibniz Institute for Educational Trajectories, Bamberg 96047, Germany
-
Silber, Henning
ORCID
Institute for Social Research, University of Michigan, Ann Arbor MI 48109, USA
-
Sleegers, Willem
ORCID
Department of Social Psychology, Tilburg University, Tilburg 5037AB, Netherlands
-
Sonntag, Nico
ORCID
Institute of Sociology, Johannes Gutenberg University Mainz, Mainz 55128, Germany
-
Staudt, Alexander
void
-
Steiber, Nadia
ORCID
Department of Sociology, University of Vienna, Vienna 1090, Austria
-
Steiner, Nils D.
ORCID
Institute for Political Science, Johannes Gutenberg University Mainz, Mainz 55099, Germany
-
Sternberg, Sebastian
ORCID
Independent Researcher
-
Stiers, Dieter
ORCID
Centre for Political Science Research, KU Leuven, Leuven 3000, Belgium
-
Stojmenovska, Dragana
ORCID
Department of Sociology, University of Amsterdam, Amsterdam 1012 WP, Netherlands
-
Storz, Nora
ORCID
Expert Council on Intergration and Migration, Berlin 10178, Germany
-
Striessnig, Erich
ORCID
Department of Sociology, University of Vienna, Vienna 1090, Austria
-
Stroppe, Anne-Kathrin
ORCID
Survey Data Curation, GESIS—Leibniz Institute for the Social Sciences, Mannheim 50667, Germany
-
Suchow, Jordan W.
ORCID
School of Business, Stevens Institute of Technology, Hoboken 07030, USA
-
Teltemann, Janna
ORCID
Institute for Social Sciences, University of Hildesheim, Hildesheim 31141, Germany
-
Tibajev, Andrey
ORCID
Department of Women's and Children's Health, Uppsala University, Uppsala SE-751 05, Sweden
-
Tung, Brian
ORCID
Department of Sociology, Washington University in St. Louis, St. Louis 63130, USA
-
Vagni, Giacomo
ORCID
Social Research Institute (UCL), University College London, London WC1E 6BT, USA
-
Van Assche, Jasper
ORCID
Center for Social and Cultural Psychology (CESCUP), Université Libre de Bruxelles, Brussels, BE-1050, Belgium; Optentia Research Unit, North-West University, Potchefstroom, 2531, South Africa
-
van der Linden, Meta
ORCID
Department of Interdisciplinary Social Science, Utrecht University, Utrecht 3584CH, Netherlands
-
van der Noll, Jolanda
ORCID
Department of Psychology, University of Hagen, Hagen 58097, Germany
-
Van Hootegem, Arno
ORCID
Department of Sociology and Human Geography, University of Oslo, Oslo 851, Norway
-
Vogtenhuber, Stefan
ORCID
Education and Employment, Institute for Advanced Studies, Vienna 1080, Austria
-
Voicu, Bogdan
ORCID
Research Institute for Quality of Life, Romanian Academy, Bucharest, 010071, Romania; Department of Sociology, Lucian Blaga University of Sibiu, Sibiu, 550024, Romania
-
Wagemans, Fieke
ORCID
Beleidsvisies, Burgervisies en Gedragingen (Policy Perspectives, Citizen Perspectives, and Behaviors), Netherlands Institute for Social Research, The Hague 2594, Netherlands
-
Wehl, Nadja
ORCID
Research Cluster ‘The Politics of Inequality’, University of Konstanz, Konstanz 78464, Germany
-
Werner, Hannah
ORCID
Department of Political Science, University of Zurich, Zürich 8050, Switzerland
-
Wiernik, Brenton M.
ORCID
Independent Researcher
-
Winter, Fabian
ORCID
Mechanisms of Normative Change, Max-Planck-Institute for Research on Collective Goods, Bonn 53113, Germany
-
Wolf, Christof
ORCID
GESIS Leibniz-Institute for the Social Science & University of Mannheim, Mannheim 68159, Germany
-
Wu, Cary
ORCID
Department of Sociology, York University, Toronto M3J 1P3, Canada
-
Yamada, Yuki
ORCID
Faculty of Social Sciences, Economics, and Business Administration, University of Bamberg, Bamberg 96052, Germany
-
Zakula, Björn
ORCID
void
-
Zhang, Nan
ORCID
Faculty of Arts and Science, Kyushu University, Fukuoka 819-0395, Japan
-
Ziller, Conrad
ORCID
Department of Political Science, University of Duisburg-Essen, Duisburg-Essen 47057, Germany
-
Zins, Stefan
ORCID
Institute for Employment Research, Federal Employment Agency, Nuremberg 90478, Germany
-
Żółtak, Tomasz
ORCID
Department of Computational Social Sciences, Institute of Philosophy and Sociology of the Polish Academy of Sciences, Warsaw 00-330, Poland
-
Nguyen, Hung H.V.
ORCID
Political Science, University of Bremen, Bremen 28359, Germany
Show more…
Published in:
- Royal Society Open Science. - The Royal Society. - 2025, vol. 12, no. 3
English
This study investigates researcher variability in computational reproduction, an activity for which it is least expected. Eighty-five independent teams attempted numerical replication of results from an original study of policy preferences and immigration. Reproduction teams were randomly grouped into a ‘transparent group’ receiving original study and code or ‘opaque group’ receiving only a method and results description and no code. The transparent group mostly verified original results (95.7% same sign and p-value cutoff), while the opaque group had less success (89.3%). Seconddecimal place exact numerical reproductions were less common (76.9 and 48.1%). Qualitative investigation of the workflows revealed many causes of error, including mistakes and procedural variations. When curating mistakes, we still find that only the transparent group was reliably successful. Our findings imply a need for transparency, but also more. Institutional checks and less subjective difficulty for researchers ‘doing reproduction’ would help, implying a need for better training. We also urge increased awareness of complexity in the research process and in ‘push button’ replications.
-
Faculty
- Faculté des lettres et des sciences humaines
-
Department
- Département de travail social, politiques sociales et développement global
-
Language
-
-
Classification
-
Demography, sociology, statistics
-
License
-
CC BY
-
Open access status
-
gold
-
Identifiers
-
-
Persistent URL
-
https://folia.unifr.ch/unifr/documents/331500
Statistics
Document views: 8
File downloads:
- breznau-et-al-2025-the-reliability-of-replications-a-study-in-computational-reproductions.pdf: 22