Journal cover Journal topic
Geoscientific Model Development An interactive open-access journal of the European Geosciences Union
https://doi.org/10.5194/gmd-2017-97
© Author(s) 2017. This work is distributed under
the Creative Commons Attribution 3.0 License.
Model evaluation paper
01 Jun 2017
Review status
This discussion paper is under review for the journal Geoscientific Model Development (GMD).
Evaluation of Integrated Assessment Model hindcast experiments: A case study of the GCAM 3.0 land use module
Abigail C. Snyder, Robert P. Link, and Katherine V. Calvin Joint Global Change Research Institute, Pacific Northwest National Laboratory, College Park, MD 20740
Abstract. Hindcasting experiments (conducting a model forecast for a time period in which observational data is available) are rarely undertaken in the Integrated Assessment Model (IAM) community. When they are undertaken, the results are often evaluated using global aggregates or otherwise highly aggregated skill scores that mask deficiencies. We select a set of deviation based measures that can be applied at different spatial scales (regional versus global) to make evaluating the large number of variable-region combinations in IAMs more tractable. We also identify performance benchmarks for these measures, based on the statistics of the observational dataset, that allow a model to be evaluated in absolute terms rather than relative to the performance of other models at similar tasks. This is key in the integrated assessment community, where there often are not multiple models conducting hindcast experiments to allow for model intercomparison. The performance benchmarks serve a second purpose, providing information about the reasons a model may perform poorly on a given measure and therefore identifying opportunities for improvement. As a case study, the measures are applied to the results of a past hindcast experiment focusing on land allocation in the Global Change Assessment Model (GCAM) version 3.0. We find quantitative evidence that global aggregates alone are not sufficient for evaluating IAMs, such as GCAM, that require global supply to equal global demand at each time period. Additionally, the deviation measures examined in this work successfully identity parametric and structural changes that may improve land allocation decisions in GCAM. Future work will involve implementing the suggested improvements to the GCAM land allocation system identified by the measures in this work, using the measures to quantify performance improvement due to these changes, and, ideally, applying these measures to other sectors of GCAM and other land allocation models.

Citation: Snyder, A. C., Link, R. P., and Calvin, K. V.: Evaluation of Integrated Assessment Model hindcast experiments: A case study of the GCAM 3.0 land use module, Geosci. Model Dev. Discuss., https://doi.org/10.5194/gmd-2017-97, in review, 2017.
Abigail C. Snyder et al.
Abigail C. Snyder et al.
Abigail C. Snyder et al.

Viewed

Total article views: 250 (including HTML, PDF, and XML)

HTML PDF XML Total BibTeX EndNote
210 33 7 250 1 6

Views and downloads (calculated since 01 Jun 2017)

Cumulative views and downloads (calculated since 01 Jun 2017)

Viewed (geographical distribution)

Total article views: 250 (including HTML, PDF, and XML)

Thereof 250 with geography defined and 0 with unknown origin.

Country # Views %
  • 1

Saved

Discussed

Latest update: 22 Jun 2017
Publications Copernicus
Download
Short summary
Experiments conducting a model forecast for a period in which observational data is available are rarely undertaken in the Integrated Assessment Model (IAM) community. When undertaken, results are often evaluated using global aggregates that mask deficiencies. Comparing land allocation simulations in GCAM with FAO observational data from 1990–2010, we find quantitative evidence that global aggregates alone are not sufficient for evaluating IAMs with global supply constraints similar to GCAM.
Experiments conducting a model forecast for a period in which observational data is available...
Share