Journal cover Journal topic
Geoscientific Model Development An interactive open-access journal of the European Geosciences Union
doi:10.5194/gmd-2016-318
© Author(s) 2017. This work is distributed
under the Creative Commons Attribution 3.0 License.
Model description paper
01 Feb 2017
Review status
This discussion paper is under review for the journal Geoscientific Model Development (GMD).
eddy4R: A community-extensible processing, analysis and modeling framework for eddy-covariance data based on R, Git, Docker and HDF5
Stefan Metzger1, David Durden1, Cove Sturtevant1, Hongyan Luo1, Natchaya Pingintha-Durden1, Torsten Sachs2, Andrei Serafimovich2, Jörg Hartmann3, Jiahong Li4, Ke Xu5, and Ankur R. Desai5 1Battelle Ecology, 1685 38th Street, Boulder, CO 80301, USA
2GFZ German Research Centre for Geosciences, Telegrafenberg, 14473 Potsdam, Germany
3Alfred Wegener Institute – Helmholtz Centre for Polar and Marine Research, Am Handelshafen 12, 27570 Bremerhaven, Germany
4LI-COR Biosciences, 4647 Superior Street, Lincoln, NE 68504, USA
5University of Wisconsin-Madison, Dept. of Atmospheric and Oceanic Sciences, 1225 West Dayton Street, Madison, WI 53706, USA
Abstract. This study presents the systematic development of an open-source, flexible and modular eddy-covariance (EC) data processing framework. This is achieved through adopting a Development and Systems Operation (DevOps) philosophy, building on the eddy4R family of EC code packages in the R Language for Statistical Computing as foundation. These packages are community-developed via the GitHub distributed version control system and wrapped into a portable and reproducible Docker filesystem that is independent of the underlying host operating system. The HDF5 hierarchical data format then provides a streamlined mechanism for highly compressed and fully self-documented data ingest and output.

This framework is applicable beyond EC, and more generally builds the capacity to deploy complex algorithms developed by scientists in an efficient and scalable manner. In addition, modularity permits meeting project milestones while retaining extensibility with time.

The efficiency and consistency of this framework is demonstrated in the form of three application examples. These include tower EC data from first instruments installed at a National Ecological Observatory (NEON) field site, aircraft flux measurements in combination with remote sensing data, as well as a software intercomparison. In conjunction with this study, the first two eddy4R packages and simple NEON EC data products are released publicly. While this proof-of-concept represents a significant advance, substantial work remains to arrive at the automated framework needed for the streaming generation of science-grade EC fluxes.


Citation: Metzger, S., Durden, D., Sturtevant, C., Luo, H., Pingintha-Durden, N., Sachs, T., Serafimovich, A., Hartmann, J., Li, J., Xu, K., and Desai, A. R.: eddy4R: A community-extensible processing, analysis and modeling framework for eddy-covariance data based on R, Git, Docker and HDF5, Geosci. Model Dev. Discuss., doi:10.5194/gmd-2016-318, in review, 2017.
Stefan Metzger et al.
Stefan Metzger et al.
Stefan Metzger et al.

Viewed

Total article views: 235 (including HTML, PDF, and XML)

HTML PDF XML Total BibTeX EndNote
162 63 10 235 4 11

Views and downloads (calculated since 01 Feb 2017)

Cumulative views and downloads (calculated since 01 Feb 2017)

Viewed (geographical distribution)

Total article views: 235 (including HTML, PDF, and XML)

Thereof 235 with geography defined and 0 with unknown origin.

Country # Views %
  • 1

Saved

Discussed

Latest update: 23 Mar 2017
Publications Copernicus
Download
Short summary
This study presents the systematic development of the eddy4R-Docker open-source, flexible and modular eddy-covariance data processing framework. Its efficiency and consistency is demonstrated by application examples to aircraft and tower data, as well as a software cross-validation. Key improvements in accessibility, extensibility, and reproducibility build the foundation for deploying complex scientific algorithms in an efficient and scalable manner.
This study presents the systematic development of the eddy4R-Docker open-source, flexible and...
Share