Workshop: Building reproducible workflows for earth sciences

ECMWF | Reading | 14-16 October 2019

Building reproducible workflows - group photo

Full size photo

Workshop description

ECMWF's research and operational activities are computationally expensive and constantly need to scale further with new requirements and challenges. There is a high demand for the use of new mechanisms for sharing code, data, environments and making the scientific workflow more reproducible and flexible to move to new platforms. For example, for the upcoming move to ECMWF's new data centre in Bologna, Italy, we would like to explore new and robust technologies to develop reproducible workflows and how cloud computing could benefit from them.

Presentations and recordings

Monday 14 October 2019

Welcome and introduction
Claudia Vitolo (ECMWF), Florian Pappenberger (ECMWF)

PDF icon small

video icon

Reproducible workflows - Setting the scene
Stephan Siemen (ECMWF)

PDF icon small

video icon

Responding to reproducibility challenges from physics to social sciences
Ana Trisovic (IQSS, Harvard University )

PDF icon small

video icon

The important role of versioning your code
Stephan Siemen (ECMWF)

PDF icon small

video icon

Reproducibility and workflows with the Integrated Forecasting System
Andrew Bennett (ECMWF)

PDF icon small

video icon

Versioning and tracking changes of vector data
Saber Razmjooei (Lutra Consulting)

PDF icon small

video icon

Leveraging OGC standards to boost reproducibilty
Ingo Simonis (OGC)

PDF icon small

video icon

Challenges and needs of reproducible workflows of Open Big Weather and Climate data
Julia Wagemann (ECMWF)

PDF icon small

video icon

The Copernicus Climate Data Store: ECMWF’s approach to providing online access to climate data and tools
Gionata Biavati (ECMWF)

PDF icon small

video icon

Standardised data representation - power of reproducible work-flow
Marijana Crepulja (ECMWF)

PDF icon small

video icon

ECMWF Data Governance
Sebastien Villaume (ECMWF)

PDF icon small

video icon

Tuesday 15 October 2019

Recap from day 1 and remarks
Claudia Vitolo (ECMWF)

n/a

video icon

Scaling Reproducible Research with Project Jupyter
Carol Willing (Project Jupyter)

PDF icon small

video icon

Automated production of high value air quality forecasts with Pangeo, Papermill and Krontab
Peter Killick (Met Office Informatics Lab)

PDF icon small

video icon

Jupyter for Reproducible Science at Photon and Neutron Facilities
Robert Rosca (European XFEL)

PDF icon small

video icon

Design of a Generic Workflow Generator for the JEDI Data Assimilation System
Mark J. Olah (UCAR / JCSDA)

PDF icon small

video icon

Building robust and reproducible workflows with Cylc and Rose
Stuart Whitehouse (Met Office)

PDF icon small

video icon

Workflow in CESM2
Jim Edwards (National Center for Atmospheric Research USA)

PDF icon small

video icon

Reproducible science at large scale within a continuous delivery pipeline: the BSC vision
Miguel Castrillo (BSC-CNS)

PDF icon small

video icon

Space Situational Awareness - Virtual Search Environment
Marek Kubel-Grabau (Eversis)

PDF icon small

video icon

CMIP6 post-processing workflow at the Met Office
Stephen Haddad (Met Office)

PDF icon small

video icon

Reproducible workflows for big data with Python
Iain Russell (ECMWF)

PDF icon small

video icon

Using containers for reproducible results
Carlos Valiente (ECMWF)

PDF icon small

video icon

Wednesday 16 October 2019

Recap from day 2 and remarks
Stephan Siemen (ECMWF)

n/a

video icon

Publishing Reproducible Geoscientific Papers: Status quo, benefits, and opportunities
Markus Konkol (University of Münster, Institute for Geoinformatics)

PDF icon small

video icon

DARE: Integrating solutions for Data-Intensive and Reproducible Science
Alessandro Spinuso (KNMI)

PDF icon small

video icon

ECMWF's new product generation - Lessons learned from development to operations
Tiago Quintino (ECMWF)

PDF icon small

video icon

Using Cloud to Streamline R&D Workflow
Roope Tervo (Finnish Meteorological Institute)

PDF icon small

video icon

A journey into the long white Cloud
Andy Ziegler (Meteorological Service NZ Ltd.)

PDF icon small

video icon

From loose scripts to ad-hoc reproducible workflows: a methodology using ECMWF's ecflow
Damien Decremer (ECMWF)

PDF icon small

video icon

The Role of Containers in Reproducible Workflows
Shane Canon (Lawrence Berkeley National Lab)

PDF icon small

video icon

Reproducing new and old operational systems on development workstations using containers
Tom Gale (Bureau of Meteorology)

PDF icon small

video icon

Remote presentation: Developing a Unified Workflow for Convection Allowing Applications of the FV3
Benjamin Blake (IMSG and NOAA/NWS/NCEP/EMC)

PDF icon small

video icon

Scaling Machine Learning with the help of Cloud Computing
Salman Iqbal (ONS / Learnk8s)

PDF icon small

video icon

ESoWC: A Machine Learning Pipeline for Climate Science
Thomas Lees (University of Oxford), Gabriel Tseng (will present remotely) (Okra Solar)

PDF icon small

video icon

A reproducible flood forecasting case study using different machine learning techniques
Sebastian Lehner (Zentralanstalt für Meteorologie und Geodynamik)

PDF icon small

video icon

Remote presentation: Singularity Containers
David Trudgian (Sylabs Inc.)

PDF icon small

video icon

Closing   video icon