Journal cover Journal topic
Geoscientific Model Development An interactive open-access journal of the European Geosciences Union
Journal topic

Journal metrics

Journal metrics

  • IF value: 5.154 IF 5.154
  • IF 5-year value: 5.697 IF 5-year
    5.697
  • CiteScore value: 5.56 CiteScore
    5.56
  • SNIP value: 1.761 SNIP 1.761
  • IPP value: 5.30 IPP 5.30
  • SJR value: 3.164 SJR 3.164
  • Scimago H <br class='hide-on-tablet hide-on-mobile'>index value: 59 Scimago H
    index 59
  • h5-index value: 49 h5-index 49
Discussion papers
https://doi.org/10.5194/gmd-2019-132
© Author(s) 2019. This work is distributed under
the Creative Commons Attribution 4.0 License.
https://doi.org/10.5194/gmd-2019-132
© Author(s) 2019. This work is distributed under
the Creative Commons Attribution 4.0 License.

Development and technical paper 18 Jun 2019

Development and technical paper | 18 Jun 2019

Review status
This discussion paper is a preprint. It is a manuscript under review for the journal Geoscientific Model Development (GMD).

An Offline Framework for High-dimensional Ensemble Kalman Filters to Reduce the Time-to-solution

Yongjun Zheng, Clément Albergel, Simon Munier, Bertrand Bonan, and Jean-Christophe Calvet Yongjun Zheng et al.
  • CNRM, Université de Toulouse, Météo-France, CNRS, Toulouse, France

Abstract. The high computational resources and the time-consuming IO (Input/Output) are major issues in offline ensemble- based high-dimentional data assimilation systems. Bearing these in mind, this study proposes a sophisticated dynamically running job scheme as well as an innovative parallel IO algorithm to reduce the time-to-solution of an offline framework for high-dimensional ensemble Kalman filters. The dynamically running job scheme runs as many tasks as possible within a single job to reduce the queuing time and minimize the overhead of starting/ending a job. The parallel IO algorithm reads or writes non-overlapping segments of multiple files with an identical structure to reduce the IO times by minimizing the IO competitions and maximizing the overlapping of the MPI (Message Passing Interface) communications with the IO operations. Results based on sensitive experiments shown that the proposed parallel IO algorithm can significantly reduce the IO times and has a very good scalability, too. Based on these two advanced techniques, the offline and online modes of ensemble Kalman filters are built based on PDAF (Parallel Data Assimilation Framework) to comprehensively assess their efficiencies. It can be seen from the comparisons between the offline and online modes that the IO time only accounts for a small fraction of the total time with the proposed parallel IO algorithm. The queuing time might be less than the running time in a low-loaded supercomputer such as in an operational context but the offline mode can be nearly as fast as, if not faster than, the online mode in terms of time-to-solution. However, the queuing time is dominant and several times larger than the running time in a high-loaded supercomputer. Thus, the offline mode is substantially faster than the online mode in terms of time-to-solution, especially for large-scale assimilation problems. From this point of view, it suggests that an offline ensemble Kalman filter with an efficient implementation and a high performance parallel file system should be preferred over its online counterpart for the intermittent data assimilation in many situations.

Yongjun Zheng et al.
Interactive discussion
Status: open (until 13 Aug 2019)
Status: open (until 13 Aug 2019)
AC: Author comment | RC: Referee comment | SC: Short comment | EC: Editor comment
[Subscribe to comment alert] Printer-friendly Version - Printer-friendly version Supplement - Supplement
Yongjun Zheng et al.
Model code and software

Offline and online EnKFs built on the top of PDAF Y. Zheng https://doi.org/10.5281/zenodo.2703420

Yongjun Zheng et al.
Viewed  
Total article views: 188 (including HTML, PDF, and XML)
HTML PDF XML Total BibTeX EndNote
134 52 2 188 1 0
  • HTML: 134
  • PDF: 52
  • XML: 2
  • Total: 188
  • BibTeX: 1
  • EndNote: 0
Views and downloads (calculated since 18 Jun 2019)
Cumulative views and downloads (calculated since 18 Jun 2019)
Viewed (geographical distribution)  
Total article views: 151 (including HTML, PDF, and XML) Thereof 151 with geography defined and 0 with unknown origin.
Country # Views %
  • 1
1
 
 
 
 
Cited  
Saved  
No saved metrics found.
Discussed  
No discussed metrics found.
Latest update: 20 Jul 2019
Publications Copernicus
Download
Short summary
This study proposes a sophisticated dynamically running job scheme as well as an innovative parallel IO algorithm to reduce the time-to-solution of an offline framework for high-dimensional ensemble Kalman filters. The offline and online modes of ensemble Kalman filters are built to comprehensively assess their time-to-solution efficiencies. The offline mode is substantially faster than the online mode in terms of time-to-solution, especially for large-scale assimilation problems.
This study proposes a sophisticated dynamically running job scheme as well as an innovative...
Citation