Global Gridded Crop Model evaluation: benchmarking, skills, deficiencies and implications
Christoph Müller1, Joshua Elliott2,3,24, James Chryssanthacopoulos3,24, Almut Arneth4, Juraj Balkovic5,6, Philippe Ciais7, Delphine Deryng2,3,24, Christian Folberth5,8, Michael Glotter9, Steven Hoek10, Toshichika Iizumi11, Roberto C. Izaurralde12,13, Curtis Jones12, Nikolay Khabarov5, Peter Lawrence14, Wenfeng Liu15, Stefan Olin16, Thomas A. M. Pugh4,17, Deepak Ray18, Ashwan Reddy12, Cynthia Rosenzweig3,19,24, Alexander C. Ruane3,19,24, Gen Sakurai11, Erwin Schmid20, Rastislav Skalsky5, Carol X. Song21, Xuhui Wang7,22, Allard de Wit10, and Hong Yang15,231Potsdam Institute for Climate Impact Research, 14473 Potsdam, Germany 2University of Chicago and ANL Computation Institute, Chicago, IL 60637, USA 3Columbia University Center for Climate Systems Research, New York, NY 10025, USA 4Karlsruhe Institute of Technology, IMK-IFU, 82467 Garmisch-Partenkirchen, Germany 5International Institute for Applied Systems Analysis, Ecosystem Services and Management Program, 2361 Laxenburg, Austria 6Comenius University in Bratislava, Department of Soil Science, 842 15 Bratislava, Slovak Republic 7Laboratoire des Sciences du Climat et de l'Environnement, CEA CNRS UVSQ Orme des Merisiers, F-91191 Gif-sur-Yvette, France 8Department of Geography, Ludwig Maximilian University, 80333 Munich, Germany 9University of Chicago, Department of the Geophysical Sciences, Chicago, IL 60637, USA 10Alterra Wageningen University and Research Centre, Earth Observation and Environmental Informatics, 6708PB Wageningen, Netherlands 11National Agriculture and Research Organization, National Institute for Agro-Environmental Sciences, Agro-Meteorology Division, Tsukuba, 305-8604, Japan 12University of Maryland, Department of Geographical Sciences, College Park, MD 20742, USA 13Texas A&M University, Texas AgriLife Research and Extension, Temple, TX 76502, USA 14National Center for Atmospheric Research, Earth System Laboratory, Boulder, CO 80307, USA 15Eawag, Swiss Federal Institute of Aquatic Science and Technology, CH-8600 Duebendorf, Switzerland 16Department of Physical Geography and Ecosystem Science, Lund University, 223 62 Lund, Sweden 17School of Geography, Earth & Environmental Science and Birmingham Institute of Forest Research, University of Birmingham, Edgbaston, Birmingham, B15 2TT, United Kingdom 18Institute on the Environment, University of Minnesota, Saint Paul, USA 19National Aeronautics and Space Administration Goddard Institute for Space Studies, New York, NY 10025, USA 20University of Natural Resources and Life Sciences, Institute for Sustainable Economic Development, 1180 Vienna, Austria 21Rosen Center for Advanced Computing, Purdue University, West Lafayette, Indiana, USA 22Peking University, Sino-French Institute of Earth System Sciences, 100871 Beijing, China 23Department of Environmental Sciences, University of Basel, Petersplatz 1, CH-4003 Basel, Switzerland 24NASA Goddard Institute for Space Studies, New York, NY 10025, USA
Received: 01 Aug 2016 – Accepted for review: 17 Sep 2016 – Discussion started: 20 Sep 2016
Abstract. Crop models are increasingly used to simulate crop yields at the global scale, but there so far is no general framework on how to assess model performance. We here evaluate the simulation results of 14 global gridded crop modeling groups that have contributed historic crop yield simulations for maize, wheat, rice and soybean to the Global Gridded Crop Model Intercomparison (GGCMI) of the Agricultural Model Intercomparison and Improvement Project (AgMIP). Simulation results are compared to reference data at global, national and grid cell scales and we evaluate model performance with respect to time series correlation, spatial correlation and mean bias. We find that GGCMs show mixed skill in reproducing time-series correlations or spatial patterns at the different spatial scales. Generally, maize, wheat and soybean simulations of many GGCMs are capable of reproducing larger parts of observed temporal variability (time series correlation coefficients (r) of up to 0.888 for maize, 0.673 for wheat and 0.643 for soybean at the global scale) but rice yield variability cannot be well reproduced by most models. Yield variability can be well reproduced for most major producer countries by many GGCMS and for all countries by at least some. A comparison with gridded yield data and a statistical analysis of the effects of weather variability on yield variability shows that the ensemble of GGCMs can explain more of the yield variability than an ensemble of regression models for maize and soybean, but not for wheat and rice. We identify future research needs in global gridded crop modeling and for all individual crop modeling groups. In the absence of a purely observation-based benchmark for model evaluation, we propose that the best performing crop model per crop and region establishes the benchmark for all others, and modelers are encouraged to investigate how crop model performance can be increased. We make our evaluation system accessible to all crop modelers so that also other modeling groups can test their model performance against the reference data and the GGCMI benchmark.
Müller, C., Elliott, J., Chryssanthacopoulos, J., Arneth, A., Balkovic, J., Ciais, P., Deryng, D., Folberth, C., Glotter, M., Hoek, S., Iizumi, T., Izaurralde, R. C., Jones, C., Khabarov, N., Lawrence, P., Liu, W., Olin, S., Pugh, T. A. M., Ray, D., Reddy, A., Rosenzweig, C., Ruane, A. C., Sakurai, G., Schmid, E., Skalsky, R., Song, C. X., Wang, X., de Wit, A., and Yang, H.: Global Gridded Crop Model evaluation: benchmarking, skills, deficiencies and implications, Geosci. Model Dev. Discuss., doi:10.5194/gmd-2016-207, in review, 2016.