Reliable numerical simulation plays a critical role in climate change study. The reliability includes bitwise identical reproducibility, i.e. bitwise identical result of numerical simulation can be reproduced. It is important to Earth system modeling and has already been used intra modeling groups for the model development. However, it is rarely considered in a wider range even worldwide. To help achieve the worldwide bitwise identical reproducibility, we introduce the detailed implementations for the bitwise identical reproducibility on the Community Coupler (C-Coupler) platform, a uniform runtime software environment that configures, builds and runs the models in the same manner. Moreover, we share a series of experiences and suggestions regarding the bitwise identical reproducibility. We believe that these implementations, experiences and suggestions can be easily extended to other model software platforms and can prospectively advance the model development and scientific researches in the future.
Earth system modeling plays a critical role in understanding the past, present, and future of the climate. With the rapid advancement of science and technology, a number of numerical models have been developed for Earth system modeling, including component models (e.g., atmosphere models, land surface models, ocean models, sea ice models and wave models), and coupled models consisting of multiple component models, such as climate system models (CSMs) and Earth system models (ESMs).
To help the development of Earth system modeling and scientific researches on climate studies, an increasing number of international model intercomparison projects (MIPs) have been proposed, which attract more and more modeling groups to participate in. Each MIP usually provides a set of standard simulation experiments for running, evaluating, understanding and improving the models. The simulation results from MIPs are always open to the whole world for climate studies. One typical example is the Coupled Model Intercomparison Project (CMIP). In the CMIP Phase 3 (CMIP3), there were less than 30 coupled model versions, while there are more than 50 coupled model versions in CMIP5. Now the CMIP5 simulation results can be freely downloaded and have been widely used for various scientific researches.
Reproducibility is a fundamental principle of scientific
researches. To improve the reproducibility of published results,
journals such as the
However, it has been shown that the availability of the code as well as input data alone cannot completely guarantee the reproduction of climate simulation results, for example mean climate states, variability and trends at different scales may be significantly changed or even lead to opposing results due to a slight change in the simulation setting during a reproduction, and therefore bitwise identical reproducibility that guarantees the reproduction of exactly the same results is important to Earth system modeling (Liu et al., 2015, Supplement 1). Moreover, the way of testing reproducibility of the simulations in a paper by referees will introduce a new burden to referees and will be impractical when the simulations require a large amount of computing resource or a long time to be finished.
Some modeling groups have already used bitwise identical reproducibility to test the models (Easterbrook and Johns, 2009). Bitwise identical tests are useful for checking that a change does not break anything it shouldn't. As they are taken very strictly and easily, they can be conducted automatically. Moreover, a short simulation time, such as several simulation days, is enough for bitwise identical tests. Therefore, bitwise identical tests can be used in regression testing, to conveniently detect errors in the model code. With increasing collaborations between different modeling groups, bitwise identical reproducibility become an important concern when developing a new model software platform (a runtime software environment for configuring, building and running the models) (Ford et al., 2012).
The bitwise identical reproducibility of Earth system modeling deserves to be a worldwide standard (Liu et al., 2015, Supplement 1). Such a standard, which means any fellow scientists can independently and conveniently reproduce published simulation results for further research, can promote the sharing of the model code, input data, knowledge and experiences, upgrade the quality and reliability of the model code (Easterbrook and Johns, 2009; Clune and Rood, 2011; Easterbrook, 2014), enable the reproducibility of the simulations in a paper to be checked automatically, and improve the trust of published simulation results. Even when the reproduction is not necessarily at a bitwise identical level, or the bitwise identical reproduction fails due to the lack of an appropriate computing environment (including parallel setting, compiler version, compiling flags, processor version, third-party libraries and operating system, Ford et al., 2012), fellow scientists are still able to conveniently and independently repeat the simulations for further research.
However, a recent survey with hundreds of published papers reveals that the bitwise identical reproducibility of Earth system modeling is currently at a very low level (Liu et al., 2015, Supplement 1). Journals and papers rarely emphasize the bitwise identical reproducibility of published simulation results. Regarding existing MIPs, almost none of them emphasize the bitwise identical reproducibility of submitted simulation results. As a result, the reproduction of an original simulation by fellow scientists heavily depends on interactions with the original authors. The original authors are often unable to help the reproduction. One typical reason is that the original simulation setting is no longer preserved due to the limited storage space or due to the rapid upgrade of computer software and hardware. Even when the original simulation setting is still preserved, the original authors may have to spend a lot of time to help the reproduction.
To achieve a bitwise identical reproduction, a straightforward and practical solution is to recreate exactly the same whole simulation setting, including the model code, input parameters, input data and computing environment (Ford et al., 2012). Intra a modeling group, as the group members always work on the same computers for the model development and scientific researches, it is unnecessary to specifically record the whole simulation setting of a simulation and bitwise identical reproducibility therefore can be easily achieved. However, for the worldwide bitwise identical reproducibility, the whole simulation setting needs to be recorded, so that a series of challenges require to be resolved.
The first challenge is that new heavy burdens would be introduced to scientists. As bitwise results are determined by the whole simulation setting (including the model code, input data, parameter setting, computing environment, etc., Ford et al., 2012) and are very sensitive to round-off errors resulting from the computing environment (Monniaux, 2008; Song et al., 2012; Hong et al., 2013), it requires original scientists to preserve the whole simulation setting and fellow scientists to recreate almost exactly the same simulation setting during a reproduction.
The second challenge is how to enable fellow scientists to conveniently obtain the whole simulation setting of published simulation results. As the whole simulation setting includes a lot of seemingly uninteresting information and is generally of a large size, it is not feasible to be detailed in a published paper or included as a Supplement.
The third challenge is permissions for distributing model code and input data files. The model code and input data files used in a simulation are generally origin from different modeling groups, with protection of copyrights. Scientists may not be permitted to publicly distribute the model code and input data files on a server beyond the modeling groups even when the model code and input data files are publicly open.
The fourth challenge is the inheritance of reproducibility. A new simulation by fellow scientists is always conducted through new modifications to an original simulation that has been successfully reproduced or repeated. Thus a part of the simulation setting of the new simulation should be inherited from the original simulations. For the worldwide bitwise identical reproducibility, the new simulation requires to be inherently reproducible.
The fifth challenge is unavailability of the original computing environment due to the rapid upgrade of computer software and hardware. When fellow scientists lack of the original computing environment for the reproduction, bitwise identical reproduction will possibly fail, and the simulation can only be repeated.
In this work, we investigate how to address the above challenges on
the Community Coupler (C-Coupler) platform (Liu et al.,
2014). C-Coupler is a Chinese community coupler for Earth system
modeling. It contains a library with functions for coupling a number
of component models together and a uniform model software platform
called the C-Coupler platform that configures, builds and runs the
models in the same manner. The first version (C-Coupler1) was finished
and publicly released recently with the enhancement for the bitwise
identical reproducibility on the C-Coupler platform. It has been used
to construct several coupled model versions mainly developed by
several institutions in China. The C-Coupler platform helps to achieve
the worldwide bitwise identical reproducibility in several aspects:
When original scientists configure a simulation with a simple
command, information of the whole simulation setting is
automatically recorded into a simulation setting package, and when
fellow scientists want to reproduce or repeat a simulation by
original scientists, a simple command can be used to automatically
download the whole simulation setting. The output data files of
a simulation can contain the keywords of the corresponding
simulation setting package. In other words, given an output data
file, scientists are able to find out the corresponding simulation
setting package and then reproduce or repeat the corresponding
simulation. A simulation setting package is generally of a small enough size
to be a supplement of a published paper, so as to enable the
reproduction or repetition of the corresponding simulation by fellow
scientists to be independent of the original authors. The model code and input data files for the same simulation can
be downloaded from different servers (called simulation resource
servers in this paper), which can guarantee infringement of
copyrights of the model code and data files when distributing the
simulation setting. Information in the simulation setting package of a reproduced
simulation will be automatically transferred into a new simulation,
to make the new simulation conducted by fellow scientists inherently
reproducible.
To make the original simulation successfully bitwise identical
reproduced even when the original computing environment is
unavailable, we further share a series of experiences in terms of
bitwise identical parallelization, bitwise identical compiler version
set and bitwise identical processor version set, and also give out
a series of suggestions.
The remainder of this paper is organized as follows. Section 2 briefly introduces background and related work. Section 3 details how to enhance the bitwise identical reproducibility on the C-Coupler platform. Section 4 gives out our experiences and suggestions for the bitwise identical reproducibility. They are not limited to the C-Coupler platform and we believe that they can help other model platforms to enhance the bitwise identical reproducibility. Section 5 empirically evaluates the C-Coupler platform in term of the bitwise identical reproducibility. We discuss and conclude this work in Sect. 6.
Information of the whole simulation setting can be viewed as a kind of
metadata (data describing data) of the simulations. Accurate and
complete metadata are valuable for the identification, assessment,
usage and reproduction of the simulation results. A series of efforts
have already been made to investigate useful sets of the metadata,
where the Common Metadata for Climate Modeling Digital Repositories
(METAFOR,
Specifically for the worldwide bitwise identical reproducibility, the technology of describing and recording the whole simulation setting needs to be further studied. The metadata of the whole simulation setting, for example, should be of a small size to be able to be a supplement of a published paper, so as that fellow scientists can reproduce the simulation results according to a paper. A new simulation derived from a previous simulation that has been successfully reproduced (or repeated) should be able to be reproduced (or repeated) by fellow scientists in the future.
The C-Coupler platform (Liu et al., 2014) used in this work manages
the source code and input data for various simulations. It designs the
same manner for uniformly operating various simulations on various
hardware platforms. On the C-Coupler platform, there are four steps to
operate a simulation: “
One prospective approach to achieving the worldwide bitwise identical reproducibility is to describe the whole simulation setting in each published paper. As the whole simulation setting contains too much details most of which are not interesting to scientists, it is appropriately to be a supplement of the paper. Due to thousands of even millions of lines of the model code and generally large size of input data files for initial states and forcings, the whole simulation setting is generally too large to be a supplement. Our proposed solution to this problem is that the package for the whole simulation setting does not contain the model code and input data files but only provides the entrances for further downloading them from simulation resource servers.
A new simulation by fellow scientists is always derived from several original simulations with new modifications. Thus parts of the simulation setting of the new simulation should be inherited from the original simulations, while the other parts are the new modifications. When the new modifications include new model code or new input data files, we advise scientists to prepare a new simulation resource server for downloading them. As a result, the reproduction of a simulation possibly requires the downloading of the model code and input data files from multiple simulation resource servers that respectively belong to different modeling groups or individual scientists.
Figure 1 shows the flowchart of achieving the bitwise identical reproducibility on the C-Coupler platform. When original scientists configure a simulation, the simulation setting package is produced automatically in order to record the whole simulation setting. After compiling the model code and then running the simulation, log files and output data files are produced, which can be further used to check the bitwise identical reproduction. After obtaining a copy of the simulation setting package, fellow scientists can use a simple command to automatically download the corresponding model code and input data files from the simulation resource servers, and then can recreate the simulation setting for a reproduction. After rerunning the simulation, bitwise identical reproduction can be checked through comparing the log files or output data files of the original simulation to the reproduced simulation. Fellow scientists can further conduct a new simulation that will be inherently reproducible, through updating the reproduced simulation and providing new simulation resource servers when necessary. In other words, fellow scientists can become original scientists in the chain of reproducible simulations.
In the following context of this section, we will detail how to record the whole simulation environment, how to record the information for checking bitwise identical reproduction, how to download and then reproduce a simulation and how to achieve the inheritance of reproducibility.
The simulation setting package generated when configuring a simulation
is named as
The simulation setting package contains the information related to the model code, input data files, input parameters and computing environment. It also contains a file with log information of the configuration, including the username (the user account on the computer for configuring the model simulation), computer name, configuration time, warning reports and error reports. In the following context of this subsection, we will detail the management of the model code, input data files, input parameters and computing environment.
Version control tools, such as CVS, Subversion and GIT, have already
been used for tracking the code development of the models (Easterbrook
and Johns, 2009; Ford et al., 2012). The C-Coupler platform also
prefers version control tools for the model code management so that
the exact version of the model code can be downloaded from simulation
resource servers for the reproduction or repetition. To achieve the
worldwide bitwise identical reproducibility, we further improve the
model code management:
A separate version control system for each component model (or
library). The code versions of component models and libraries that
constitute a coupled model version are possibly downloaded from
different modeling groups. Even though the code versions can be
downloaded from the same modeling group, they may belong to
different released coupled model versions. For example, a new
coupled model version can be constructed through replacing the
atmosphere model version in the Community Climate System Model
Version 3 (Collins et al., 2006) (CCSM3) with the atmosphere model
version in the Community Earth System Model (Hurrell et al., 2013)
(CESM). To address these cases, the C-Coupler platform supports
a separate version control system for each component model (or
library). When configuring a simulation, for each component model
(or library), the code version number and the simulation resource
server for downloading the model code are detected and recorded in
the simulation setting package. When recreating the simulation
setting for a reproduction or repetition, the code version of each
component model (or library) will be checked out from the
corresponding simulation resource server. Code patching. Only the code versions that have been committed
to a version control system can be further checked out. Thus it is
possible that the model code for a simulation cannot be fully
downloaded from a simulation resource server because some code
modifications have not been committed as a new code version before
configuring the simulation. Moreover, the fellow scientists who are
not in the group of developing a component model generally do not
have permissions to either commit a code modification to the version
control system of the component model or build a new simulation
resource server for publicly distributing the modified code with
a code version of the component model. Considering these cases, we
implemented a function called code patching. When configuring
a simulation, for each component model (or library), the C-Coupler
platform will differentiate the currently used model code with the
corresponding code version managed by the version control system and
then record all differences as a code patch into the simulation
setting package. When the code of a component model (or library) is
not managed by any version control system, all the code will be
recorded as a code patch. When recreating the simulation setting for
a reproduction or repetition, the code patch will be used to recover
the code of the corresponding component model (or
library).
The above implementations guarantee that the model code used in
a simulation can be exactly recovered for the reproduction even when
the model code is not managed by version control systems. They enable
any scientists to distribute their efforts for the model development
without violating the copyrights of the model code that is obtained
from other modeling groups. Moreover, they enable a modeling group to
easily get code modifications from fellow scientists to improve the
model development.
Similar to the model code management, version control tools are also preferred for both tracking the changes of the input data files and for downloading these files. As the input data files are possibly downloaded from different simulation resource servers, the C-Coupler platform manages each input data file separately.
At the present time, Subversion is used as the default version control tool for the input data files but not other version control tools such as GIT, because we find that Subversion is much friendlier for downloading data files. When configuring a simulation, all required input data files will be linked to the working directory of the simulation, and the information of each input data file, including the file name, the address of the Subversion server, the version number, and the checksum of the file content will be detected and recorded into the simulation setting package. The checksum is used because previous works have demonstrated that it is helpful for identifying a correct input data file (Ford et al., 2012). When an input data file is not managed by Subversion, the whole data file will not be put into the simulation setting package for saving the storage space, while only the checksum is recorded.
It is time-consuming to calculate the checksum of a data file,
especially when the data file is large. We therefore provide two types
of configuration. When scientists use the command
“
On the C-Coupler platform, the files of input parameters for a simulation are generated by scripts. To modify the input parameters, scientists should modify the corresponding scripts that will be automatically recorded into a simulation setting package when configuring a simulation. Therefore, the input parameter files can be exactly regenerated in a reproduction.
As mentioned in Sect. 2, the computing environment for a simulation includes the parallel setting, compiler version, compiling flags, processor version, third-party libraries and operating system (Ford et al., 2012). At the present time, the C-Coupler platform does not take into consideration of the operating system and third-party libraries except Message Passing Interface (MPI) libraries, because almost all the models that have already been ported to the C-Coupler platform run on the Linux operating systems and rarely use the third-party libraries beyond MPI libraries.
On the C-Coupler platform, there is a configuration file for specifying the parallel setting, including the number of processes and the number of threads to run each component model. For each component model, there are configuration files to specify the compiler version (the MPI version is treated as a part of the compiler version) and compiling flags for the compilation. Each computer system has a unique name that can help scientists to find the corresponding computer system or get more information from the web. To enable the simulations to run on a computer system, there is a configuration file which specifies how to submit a simulation on that computer system. When configuring a simulation, all configuration files related to the computing environment are recorded into the simulation setting package. After modifying the computing environment, scientists should reconfigure the simulation and even recompile the model code when necessary. Currently, we only consider the homogeneous computer systems where all processors are the same. Therefore, the deployment of processors for running a simulation is not concerned yet.
The output from a simulation run, including the log files and output data files, can be provided for checking whether bitwise identical reproduction is achieved. If the model iteratively diagnoses some global masses during a simulation, the log files alone can be enough for the checking. As the output from a short-time simulation is enough for bitwise identical tests (Easterbrook and Johns, 2009), it is practical to keep the output in the supplement of a paper for bitwise identical tests.
On the C-Coupler platform, the log files from a simulation run are
named with the
Given a simulation setting package, the model code and input data
files of the corresponding simulation can be downloaded with a simple
command “
After downloading a simulation, fellow scientists can reproduce or
repeat the simulation with the following steps:
Create a working directory for the reproduced simulation and
then unpack the corresponding simulation setting package under this
directory. Link the simulation to the directories of the downloaded model
code and input data files, through setting environmental variables. The default parallel setting is the same as that recorded in the
simulation setting package. If the corresponding model version can
achieve bitwise identical result in different parallel settings, the
parallel setting can be modified flexibly. Verify the compiler version, and then try to install the
corresponding compiler version if necessary; verify the processor
version. Section 4 will discuss about the compiler version set and
processor version set that can achieve bitwise identical result. Configure the simulation, compile the model code, and then run
the simulation. Check whether the bitwise identical result is reproduced. The
log files, output data files and figures from the original model
simulation can help this check. If the bitwise identical result is not reproduced, please
contact the authors of the original model simulation if
necessary. One possible cause for this situation is that there are
bugs in the model code.
Generally, it is very strict to reproduce the bitwise identical
result. When the local computer lacks of the appropriate compiler
versions and processor versions, the bitwise identical result is
hardly reproduced. However, the simulation can be repeated on the
local computer with the above steps, and then the newly derived
simulations can achieve the bitwise identical reproducibility on the
local computer.
To make simulations inherently reproducible, we first make the C-Coupler platform inherently reproducible. The C-Coupler platform is treated as a part of the whole simulation setting, and also managed by a version control system. Its version number and code patch are recorded into the simulation setting package when configuring a simulation. When downloading a simulation, the C-Coupler platform will be downloaded and then recovered with the corresponding code patch. Thus for a new simulation derived from a reproduced original simulation, the C-Coupler platform is still available for achieving the bitwise identical reproducibility. Second, the simulation setting package tracks the history information of simulation resource servers for each input data file and the code of each component model (or library). A new simulation will inherit the history information from the original simulation. As a result, the model code and input data files of the new simulation that are inherited from the original simulation can also be downloaded from the simulation resource servers of the original simulation. Third, the new modifications in a new simulation beyond the original simulation can be recorded in the simulation setting package or be downloaded from new simulation resource servers newly specified by fellow scientists.
The above implementations on the C-Coupler platform help achieve the worldwide bitwise identical reproducibility. The automatic generation of the simulation setting package does not introduce extra works to scientists when conducting a simulation. The model code and input data files will be automatically downloaded and recovered when reproducing a simulation, which facilitates the reproduction of a simulation. Through combining the version control tools for downloading the model code and input data files, the simulation setting package is generally small enough to be a supplement of a published paper. Through supporting multiple simulation resource servers and code patching, any scientist can directly contribute to the model development and cooperate with the modeling groups, without infringement of copyrights of the model code and data files. Through making the C-Coupler platform inherently reproducible and tracking the history information of simulation resource servers, a new simulation derived from a successfully reproduced simulation can be inherently reproducible.
The above implementations are enough for the models already on the
C-Coupler platform. Targeting the bitwise identical reproducibility of
more models, the following aspects need to be considered in the
future:
The information of the third-party libraries and the operating
system for a simulation should be managed and recorded in the
simulation setting package. More kinds of tools for managing and downloading the input data
files should be supported. Databases that allow the query of
information deserve to be considered for the input data file
management. The API for writing the keywords of a simulation setting package
into the output data files of a simulation should support more kinds
of data file formats, not only the NETCDF format.
It often happens that a simulation several years ago cannot be reproduced at a bitwise identical level because the original computing environment is unavailable due to the rapid upgrade of computer software and hardware. Therefore, it is valuable to study how to make bitwise identical reproduction achieved on different computing environments. In this section, we share our relevant experiences, including how to make the parallelization of the models achieve bitwise identical result in different parallel settings, and which compiler versions and processor versions can produce bitwise identical result. The parallelization that achieves bitwise identical result in different parallel settings is called bitwise identical parallelization, the set of compiler versions that achieve bitwise identical result is called bitwise identical compiler version set, and the set of processor versions that achieve bitwise identical result is called bitwise identical processor version set. Moreover, we give some suggestions for facilitating achievement of the bitwise identical reproducibility.
The C-Coupler platform does not require the models to achieve the bitwise identical parallelization, because the parallel setting of a simulation is recorded in the simulation setting package. However, we highlight the bitwise identical parallelization due to two reasons. First, it is a strict and valuable testing standard for correct parallelization. Wrong parallelization generally cannot guarantee bitwise identical result in different parallel settings. If a parallel version is allowed to produce different results when the parallel setting is modified, it will be difficult to verify whether the parallelization is correct. Second, with bitwise identical parallelization, scientists can freely change the parallel setting when reproducing a simulation. If the computing resource is limited, scientists can use a smaller number of processor cores to reproduce the simulation.
Actually, bitwise identical parallelization has already been
considered for the model development
( Compiling flags. Different compiler optimizations may lead to
different calculation orders of floating-point operations, different
round-off errors, and finally different simulation results. We find
that, bitwise identical parallelization is relevant to compiler
optimizations, and the optimization level O0, which keeps the
original floating-point calculation orders in the model code, is
more helpful for achieving bitwise identical parallelization. For
higher optimization levels, scientists are advised to find out the
compiling flags which can guarantee the original calculation
orders. For example, regarding the Intel compilers, the set of
compiling flags “-no-vec -mp1 -fp-model precise -fp-speculation Global summation operations. There are always global summation
operations in the model code. Scientists can use the reduce function
in the Message Passing Interfaces (MPI) library to achieve global
summation in parallel. However, this implementation possibly results
in non-bitwise identical parallelization because the MPI reduce
function will change the order of the floating-point calculations
and introduce different round-off errors when using different
parallel settings. To achieve bitwise identical global summation, we
propose two approaches. First, one process can be selected to gather
all data values and calculate the sum, and then broadcasts the sum
to all processes if required. Second, the MPI reduce function can be
used to calculate summation with higher precision than that of the
corresponding floating-point values. For example, for
single-precision floating-point values, double-precision summation
should be used, and for double-precision values,
long-double-precision (128-bit wide) summation should be
used. Although higher-precision summation may introduce higher
computation cost, especially for the long-double-precision
summation, we prefer the second approach because it performs the
summation in parallel and the corresponding communication overhead
is much smaller than the first approach.
It is unnecessary to use exactly the same compiler version and
processor version for reproducing the bitwise identical result of
a simulation. For example the Intel Fortran and C/C
Note that the Intel
To facilitate the achievement of bitwise identical reproducibility, we would like to give a series of suggestions in several aspects, including the model code, input data files, compiler, processor, simulation setting package and bitwise identical tests.
Modeling groups usually use a version control system to track the development of the model code. Currently, popular version control tools include Subversion and GIT. GIT is used as the default version control tool for the model code managed on the C-Coupler platform. We prefer GIT rather than Subversion because GIT is a distributed version control tool, which can facilitate multiple institutions to cooperatively develop a model. Moreover, GIT provides more functions for version control, such as branch version, local commits, etc. Indeed, GIT has already been used for the model development (Jacobsen et al., 2013).
When the model code is not managed by a version control system, the
bitwise identical reproducibility can also be achieved due to the code
patching function. We do not implement a function of
It is prospective to combine a database and a version control system together for the input data file management, where the database can be used to query the information of the input data files.
When conducting a simulation, the compiler and processor should be
concerned. If the bitwise identical result of a simulation highly
depend on a specific compiler version and a specific processor
version, it will be inconvenient for fellow scientists to reproduce
the simulation. For example, scientists may have to find a computer
with the specific processor version and install the specific compiler
version on that computer, which will introduce a lot of work to do and
even is impractical. We therefore give the following suggestions:
Use common compilers and common processors for the model
development. Common processors such as the Intel CPUs have been used
worldwide, which makes it easier to find a computer for the
reproduction. Common compilers such as the GNU compilers and Intel
compilers have often been installed on computers, which can possibly
save the work for installing a compiler version for the
reproduction. Extend the bitwise identical compiler version set and processor
version set. As introduced in Sect. 4.1.2, multiple compiler
versions can constitute a bitwise identical compiler version set and
multiple processor versions can constitute a bitwise identical
processor version set. Through extending these two sets, it will
become more convenient to reproduce the simulations. These two sets
discovered by us currently are limited to several Intel Compiler
versions and several CPU versions. In the future, we will try to
extend these two sets through investigating and including more
compilers and more processors, such as the GNU compiler versions and
the AMD CPU versions.
Given an output data file, the first step for reproducing the corresponding simulation is to find the corresponding simulation setting package, according to the log information in the output data file. Scientists are advised to preserve the simulation setting packages for various simulations. It is convenient to keep a large number of simulation setting packages on modern storage, because the size of a simulation setting package is generally small (e.g., smaller than 1 MB). As a result, fellow scientists can obtain a simulation setting package according to a published paper or an output data file. For a modeling group, there could be a shared directory to store all simulation setting packages provided by different scientists. Version control tools or databases can be used to effectively manage the simulation setting packages.
Bitwise identical tests have already been used for the model
development
( Bitwise identical result in the repeated runs under the same
simulation setting. The bitwise identical reproducibility requires
a simulation achieving the bitwise identical result in the repeated
runs under the same simulation setting. If the bitwise identical
result is not obtained, it is very likely that there are bugs in the
corresponding model code, such as uninitialized variables and
out-of-bounds array accesses. Following the development of physical
parameterization schemes, random numbers may be introduced to the
descriptions of some stochastic processes in the schemes. To
guarantee the bitwise identical reproducibility, the random numbers
must be able to be reproduced. One choice is to use pseudorandom
numbers, a deterministic sequence of numbers that approximate the
properties of random numbers. Pseudorandom numbers are determined by
a small set of initial values. When the initial values are treated
as input parameters, the pseudorandom numbers can be fully
reproduced. Bitwise identical result on different computing
environments. Given a bitwise identical compiler version set with
Worldwide bitwise identical tests. If more and more simulations
in published papers can be reproduced, they can be used by modeling
groups for bitwise identical tests. Then modeling groups can obtain
much more test cases for the model development. Moreover, the fellow
scientists who want to reproduce the simulations in published papers
and MIPs will also contribute to bitwise identical tests.
To empirically evaluate the enhancement for the bitwise identical reproducibility on the C-Coupler platform, we use two coupled model versions that are already on the C-Coupler platform, i.e., FGOALS-gc and MASNUM-POM. FGOALS-gc is a modified version based on FGOALS-g2 (Li et al., 2013a), a CMIP5 CSM, where the original coupler CPL6 (Craig et al., 2005) used in FGOALS-g2 is replaced by C-Coupler1. MASNUM-POM is a coupled model version consisting of a wave model MASNUM (Yang et al., 2005) and a parallel version of the ocean model POM (Wang et al., 2010).
We use four computer platforms for the evaluation. Table 1 shows the
processor version and Operating System (OS) version on each computer
platform. Although all of these computer platforms use the Intel CPU
and Linux OS, the versions of the processor and OS are different. As
introduced in Sect. 4.1.2, the Intel CPU versions on these computer
platforms belong to the same bitwise identical processor version
set. On each computer platform, we installed three versions of the
Intel compiler, including Version 12.1.3, 13.0.0 and 14.0.2, which belong
to the same bitwise identical compiler version set. In each
compilation of the model code, we add “-no-vec -mp1
-fp-model precise -fp-speculation
For each coupled model version, one simulation is employed for this
evaluation. In detail, a simulation based on the CMIP5 piControl
experiment is used to run FGOALS-gc, and a simulation starting from
1 October 2012 is used to run MASNUM-POM. Actually, the evaluation for
the bitwise identical reproducibility does not depend on which
simulations are used, and short-time simulations are enough for the
evaluation. Given a simulation, we conduct the evaluation with the
following steps:
Establish a benchmark simulation run on the computer platform
No. 1 using the Intel compiler Version 12.1.3. The corresponding
simulation setting package is generated. Use the simulation setting package to reproduce the simulation
on the computer platform No. 1 using each other Intel compiler
version, e.g., Version 13.0.0 and 14.0.2. On each other computer platform, e.g., No. 2, No. 3 and No. 4,
use the simulation setting package to download the simulation from
the computer platform No. 1 (all the model code is managed by GIT
and all input data files are managed by SVN), and then reproduce the
simulation using each Intel compiler version. Check whether all reproduced simulation runs achieve the bitwise
identical result with the benchmark simulation run. There are
totally 11 reproduced simulation runs.
The evaluation result shows that, for the simulation of each coupled
model version, the 11 reproduced simulation runs achieve the bitwise
identical result with the benchmark simulation run. We therefore can
conclude that the C-Coupler platform can help the simulations to
achieve the bitwise identical reproducibility.
This paper details how to enhance the bitwise identical reproducibility of Earth system modeling on the C-Coupler platform, and give a series of experiences and suggestions accordingly. Although it only focuses on simulations of the models, the proposed implementations can be extended to pre-processes and post-processes of simulations. In the future work, we will integrate pre-process and post-process functions into the C-Coupler platform, with enhancement for the bitwise identical reproducibility of them.
The information recorded in the simulation setting package is a kind of metadata for the simulations. It can help complement the metadata developed by existing works, such as the METAFOR project. Moreover, the metadata developed by METAFOR for describing the models and their simulations can be recorded into the simulation setting package, to improve the query of simulation setting packages in a database.
The C-Coupler platform is a new software platform for the model development. There are also other software platforms which have been successfully and widely used for model development, e.g., the CCSM3 platform (Collins et al., 2006), the CCSM4/CESM platform (Gent et al., 2011; Hurrell et al., 2013), the FMS (Balaji, 2004), etc. We believe that the proposed implementations on the C-Coupler platform can be easily extended to other platforms. Moreover, we give a series of experiences and suggestions for the bitwise identical reproducibility, which are originated from our model development. We believe that they can benefit the model development in other modeling groups.
In our works to achieve the bitwise identical reproducibility, we use
the
Facilitation of the bitwise identical reproducibility also requires
the efforts beyond the field of Earth system modeling. Through
enlarging the bitwise identical compiler version set and processor
version set, the bitwise identical result of a simulation can be
reproduced more easily and flexibly. We therefore hope the experts in
the field of computer will make the new versions of compilers and
processors join in existing bitwise identical compiler version sets
and processor version sets. For example, we find that, the model
simulation result with the Intel compiler version
In the Supplement 3 of this paper, there is a package that contains
the script
We give a simulation of an atmosphere model version GAMIL2 (Li et al., 2013b) but not a simulation of FGOALS-gc or MASNUM-POM that has been used in the empirical evaluation in Sect. 5 because we cannot publicly distribute the model code without permissions from the corresponding modeling groups. GAMIL2 is the atmosphere component of FGOALS-gc.
If you encounter any problem when reproducing the given simulation, please contact us (liuli-cess@tsinghua.edu.cn or lrz04@mails.tsinghua.edu.cn). If you want to reproduce the simulations of FGOALS-gc or MASNUM-POM, please also contact us.
This work is supported in part by the Natural Science Foundation of China (no. 41275098), the National Grand Fundamental Research 973 Program of China (no. 2014CB441302) and the Tsinghua University Initiative Scientific Research Program (no. 20131089356).
Four computer platforms used in the empirical evaluation.
Flowchart for achieving the bitwise identical reproducibility on the C-Coupler platform.