The Scientific Paper of the Future (SPF) concept, initiated by the EarthCube OntoSoft Funded Project, encourages scientists to publish not only peer-reviewed journal articles, but also all associated data, software (data processing scripts), and computational workflows, in order to enable full science reproducibility. While the SPF concept was originally aimed at geoscientists, it can also be applied to interdisciplinary projects such as between ecology, economics, and maritime shipping. Multi-region input-output (MRIO) analysis is a method from economics for analyzing economic interdependencies between different regional entities. Entities can be countries, regions within a country, or groups of countries. MRIO can also be used to analyze other types of interdependencies, such as the environmental impact of one region’s activities on another. For this project, we use MRIO to analyze the global spread of marine non-indigenous species via cargo ships. Over 90% of global trade occurs by maritime shipping. Along with intended cargo, ships provide a means for marine organisms to move to locations beyond their natural ranges, mainly via hull fouling or in ballast tanks. These species can have harmful ecological and economic impacts at their destinations. By using MRIO to follow the imports and exports of commodities between countries, we can deduce the magnitude of seaborne trade connections based on physical volume of commodity traded, and therefore the magnitude and geographic distribution of marine biosecurity risk. MRIO model construction involved incorporating a diversity of data types from ecology, economics, and shipping, and has turned out to be a surprisingly complex endeavor. My poster will demonstrate the principles of an SPF by providing a diagram of the computational workflow involved in the model’s construction, including an explanation for each dataset incorporated into the model’s input parameters and each piece of software written to process the data and assemble and run the model.

The EarthCube BALTO broker (Brokered Alignment of Long-Tail Observations) provides streamlined access to both long-tail and big data using Web Services through several distinct mechanisms. First, we updated the OPeNDAP framework Hyrax, software that serves big data from USGS, NASA, and other sources, with a BALTO extension that tags dataset landing pages with JSON-LD encoding automatically. Therefore, the big data made available through Hyrax are now searchable via EarthCube GeoCODES (formerly P418) and Google Dataset Search. The BALTO broker extension to Hyrax makes thousands of datasets easily searchable and accessible. Second, we focused our efforts on a geodynamics use-case aimed at advancing our understanding of continental rifting processes through the use of an NSF mantle convection code called ASPECT. By addressing this use-case, we implemented a web services brokering capability in ASPECT that allows for remotely accessing datasets via a URL defined in an ASPECT parameter file. Third, through another use-case in ASPECT aimed at testing hypotheses involving global mantle flow, we developed a brokering mechanism for a “plug-in” that accesses NetCDF seismic tomography data from the NSF seismology facility IRIS, then transforms it into the format needed by ASPECT to run global mantle flow models constrained by seismic tomography. Fourth, we demonstrate methods to allow any scientist or citizen scientist to make their in-situ IoT based sensor data collection efforts available to the world. Finally, we are developing a Jupyter Notebook with a GUI that allows for users to search Hyrax servers for big datasets and long-tail data. These cyberinfrastructure developments comprise the entire EarthCube BALTO brokering capabilities.

As the size of geoscience datasets grows, scientists are eager to move away from a download-based workflow, where data files are downloaded a local computer for analysis, towards a more cloud-native workflow, where data is loaded on demand over the network. On-demand data loading offers several advantages, including increased reproducibility, provenance tracking, and, potentially, scalability using distributed cloud computing. In this notebook, we demonstrate how to load data on-demand using three different remote data access protocols: - OPeNDAP, the most common, well-established protocol - NetCDF over HTTP, enabled by the h5py library - Zarr over HTTP, a new format optimized for cloud object storage (e.g. Amazon S3) We then conduct a simple benchmarking exercise to explore the throughput and scalability of each service. We use Dask to parallelize reads from each access protocol and calculate the throughput as a function of number of parallel reads. One conclusion is that Zarr over HTTP, coupled with cloud object storage, shows favorable scaling up to hundreds of parallel processes. Finally, we compare the throughput of Zarr over HTTP on a few different clouds, including Google Cloud Storage, Jetstream Cloud, Wasabi Cloud, and Open Storage Network.

The Pangeo project recently introduced large parts of the CMIP6 data archive into the cloud. This enables, for the first time, centralized, reproducible science of state-of-the-art climate simulations without the need to own large storage or a supercomputer as a user. The data itself however, still presents significant challenges for analysis, one of which is applying operations across many models. Two of the major hurdles are different naming/metadata between modeling centers and complex grid layouts, particularly for the ocean components of climate models. Common workflows in the past often included interpolating/remapping desired variables and creating new files, creating organizational burden, and increasing storage requirements. We will demonstrate two Pangeo tools which enable seamless calculation of common operations like vector calculus operators (grad, curl, div) and weighted averages/integrals across a wide range of CMIP6 models directly on the data stored in the cloud. cmip6_preprocessing provides numerous tools to unify naming conventions and parse grid information and metrics (like cell area). This information is used by xgcm to enable finite volume analysis on the native model grids. The combination of both tools facilitates fast analysis while ensuring a reproducible and accurate workflow.

As more analysis-ready datasets are provided on the cloud, we need to consider how researchers access data. To maximize performance and minimize costs, we move the analysis to the data. This notebook demonstrates a Pangeo deployment connected to multiple Dask Gateways to enable analysis, regardless of where the data is stored. Public clouds are partitioned into regions, a geographic location with a cluster of data centers. A dataset like the National Water Model Short-Range Forecast is provided in a single region of some cloud provider (e.g. AWS’s us-east-1). To analyze that dataset efficiently, we do the analysis in the same region as the dataset. That’s especially true for very large datasets. Making local “dark replicas” of the datasets is slow and expensive. In this notebook we demonstrate a few open source tools to compute “close” to cloud data. We use Intake as a data catalog, to discover the datasets we have available and load them as an xarray Dataset. With xarray, we’re able to write the necessary transformations, filtering, and reductions that compose our analysis. To process the large amounts of data in parallel, we use Dask. Behind the scenes, we’ve configured this Pangeo deployment with multiple Dask Gateways, which provide a secure, multi-tenant server for managing Dask clusters. Each Gateway is provisioned with the necessary permissions to access the data. By placing compute (the Dask workers) in the same region as the dataset, we achieve the highest performance: these worker machines are physically close to the machines storing the data and have the highest bandwidth. We minimize cost by avoiding egress costs: fees charged to the data provider when data leaves a cloud region.

Computer simulations of the Earth’s climate and weather generate huge amounts of data. These data are often persisted on HPC systems or in the cloud across multiple data assets of a variety of formats (netCDF, zarr, etc…). Finding, investigating, loading these data assets into compute-ready data containers costs time and effort. The data user needs to know what data sets are available, the attributes describing each data set, before loading a specific data set and analyzing it. In this notebook, we demonstrate the integration of data discovery tools such as intake and intake-esm (an intake plugin) with data stored in cloud optimized formats (zarr). We highlight (1) how these tools provide transparent access to local and remote catalogs and data, (2) the API for exploring arbitrary metadata associated with data, loading data sets into data array containers. We also showcase the Pangeo catalog, an open source project to enumerate and organize cloud optimized climate data stored across a variety of providers, and a place where several intake-esm collections are now publicly available. We use one of these public collections as an example to show how an end user would explore and interact with the data, and conclude with a short overview of the catalog’s online presence.

ASPECT (Advanced Solver for Problems in Earth’s ConvecTion) is an analysis tool that simulates convection in the Earth’s mantle and other planets. The BALTO project has extended the ASPECT software so it can read data used to perform the simulations from the BALTO brokering server. These additions to the ASPECT codebase allow data to be remotely accessed and then processed as if the data were stored on the user’s local computer. The additions to ASPECT can be split into two distinct sections: a URL reader and a netCDF reader/translator. The URL reader uses the Data Access Protocol (DAP) to access remote data from supported web servers. Data values are transferred from the BALTO broker and converted within the URL reader plugin to match the format that is expected by the rest of the ASPECT code. Similarly, the netCDF plugin reads data stored using netCDF from the BALTO broker and transforms these data into the sph file format required by ASPECT to perform global mantle convection. The netCDF plugin can use either local or remote data. Once the NetCDF data are read, the plugin combines and formats the required variables (longitude, latitude, seismic velocity, and depth). These newly formatted values are then converted into the sph internal representation to be used as spherical harmonic data by ASPECT. The conversion and processing of data all takes place within the ASPECT program. Both plugins have been integrated to allow the user to lookup remote data in a seamless fashion and broaden the types of data that can be requested by the user.

Climate data from Earth System Models are increasingly being used to study the impacts of climate change on a broad range of biogeophysical (forest fires, fisheries, etc.) and human systems (reservoir operations, urban heat waves, etc.). Before this data can be used to study many of these systems, post-processing steps commonly referred to as bias correction and statistical downscaling must be performed. “Bias correction” is used to correct persistent biases in climate model output and “statistical downscaling” is used to increase the spatiotemporal resolution of the model output (i.e. 1 deg to 1/16th deg grid boxes). For our purposes, we’ll refer to both parts as “downscaling”. In the past few decades, the applications community has developed a plethora of downscaling methods. Many of these methods are ad-hoc collections of post processing routines while others target very specific applications. The proliferation of downscaling methods has left the climate applications community with an overwhelming body of research to sort through without much in the form of synthesis guiding method selection or applicability. Motivated by the pressing socio-environmental challenges of climate change – and with the learnings from previous downscaling efforts in mind – we have begun working on a community-centered open framework for climate downscaling: scikit-downscale. We believe that the community will benefit from the presence of a well-designed open source downscaling toolbox with standard interfaces alongside a repository of benchmark data to test and evaluate new and existing downscaling methods. In this notebook, we provide an overview of the scikit-downscale project, detailing how it can be used to downscale a range of surface climate variables such as air temperature and precipitation. We also highlight how scikit-downscale framework is being used to compare existing methods and how it can be extended to support the development of new downscaling methods.

The Magnetics Information Consortium (MagIC), hosted at http://earthref.org/MagIC is a database that serves as a Findable, Accessible, Interoperable, Reusable (FAIR) archive for paleomagnetic and rock magnetic data. It has a flexible, comprehensive data model that can accomodate most kinds of paleomagnetic data. The **PmagPy** software package is a cross-platform and open-source set of tools written in Python for the analysis of paleomagnetic data that serves as one interface to MagIC, accommodating various levels of user expertise. It is available through github.com/PmagPy. Because PmagPy requires installation of Python, several non-standard Python modules, and the PmagPy software package, there is a speed bump for many practitioners on beginning to use the software. In order to make the software and MagIC more accessible to the broad spectrum of scientists interested in paleo and rock magnetism, we have prepared a set of Jupyter notebooks, hosted on [jupyterhub.earthref.org](https://jupyterhub.earthref.org) which serve a set of purposes. 1) There is a complete course in Python for Earth Scientists, 2) a set of notebooks that introduce PmagPy (pulling the software package from the github repository) and illustrate how it can be used to create data products and figures for typical papers, and 3) show how to prepare data from the laboratory to upload into the MagIC database. The latter will satisfy expectations from NSF for data archiving and for example the AGU publication data archiving requirements.

With observational data becoming widely available, researchers struggle to find information enabling assessment for its reliable use. A small first-step toward enabling data quality assessment of observational data is to associate the data with the sensor used to make the observations and to have the sensor description machine-harvestable. In the latest additions to the X-DOMES (Cross-Domain Observational Metadata for Enviromental Sensing) toolset, we have created targeted editors for creating SensorML documents to describe sensor models. The team has adjusted its delivery to enable integration of the X-DOMES content with the GEOCODES (JSON-LD/schema.org) EarthCube project. At our poster-session, we will highlight the new changes and capabilities and demonstrate the use of new X-DOMES tools.

Many ocean and climate models output ocean variables (like velocity, temperature, oxygen concentration etc.) in depth space. Property transport in the ocean generally follows isopycnals, but isopycnals often move up and down in depth space. A small difference in the vertical location of isopycnals between experiments can cause a large apparent difference in ocean properties when the experiments are compared in depth space. As a result, it is often useful to compare ocean variables in density space. This work compares two algorithms for plotting ocean properties in density coordinates, one written in FORTRAN with a python wrapper (xlayers), and one written in python (xarrayutils). Both algorithms conserve total salt content in the vertical, and both algorithms are easily parallelizable to enable plotting large datasets in density coordinates. We apply these algorithms to plot salinity in density space in some of the CMIP-6 models. In general, areas with net precipitation today experience increasing precipitation in higher greenhouse-gas scenarios, and areas with net evaporation today experience a further reduction in net precipitation in higher greenhouse-gas scenarios. By plotting salinity in density space, we visualize how changes in evaporation and precipitation at the surface propagate along isopycnals to influence salinity concentrations in the ocean interior.

The EarthCube BALTO (Brokered Alignment of Long-Tail Observations) project is aimed at developing new cyberinfrastructures that enables brokered access to diverse geoscience datasets. Towards achieving this BALTO objective, we developed a plug-in for the community extensible NSF open-source code ASPECT (Advanced Solver for Problems in Earth’s Convection) that permits ASPECT to read data from the BALTO server (OPeNDAP’s Hyrax open-source data server) over the web. We present a use-case of the BALTO-ASPECT client, which accesses lithospheric structures from the BALTO server to constrain a 3-D lithospheric modulated convection (LMC) modeling and melt generation beneath the Rungwe Volcanic Province (RVP) and the Malawi Rift. We test the hypothesis that at least part of the melt feeding the RVP is generated from LMC. In the model, we assume a rigid lithosphere, while for the asthenosphere we use non-Newtonian, temperature-, pressure- and porosity-dependent creep laws of peridotite. We find that a significant percentage of decompression melt from LMC occurs at a maximum depth of ~200 km beneath the axis of the Malawi Rift, consistent with the location and maximum depth of imaged low velocity zones. At shallower depths (~100 km), the melting region is focused beneath the RVP where there is rapid (~3 cm/yr) upwelling. Our results suggest that asthenospheric upwelling due to LMC is the main source of melt beneath the RVP and might also entrain the plume head materials with reported high 3He/4He values. We, therefore, propose that part of the melt beneath the northern Malawi Rift feeding the RVP can be generated by LMC without necessitating plumes impinging the base of the lithosphere at present. This use-case demonstrates the capability of the BALTO-ASPECT client to accelerate research by brokering input data from the BALTO server for modeling LMC and melt generation.

We present novel applications of yt, a tool originally designed for analysis of astrophysics datasets, to the geosciences. yt is a python-based platform for volumetric data, which enables semantically meaningful analysis and visualization. As part of a wider effort to bring yt to the geosciences, we present an initial use-case of yt applied to 3D seismic tomography models of the upper mantle from the IRIS Earth Model Collaboration. While the rendering capabilities of yt can in general be applied directly to 3D geoscience data, we add several graphical elements to the 3D rendering to aid in interpretation of volume renderings including latitude/longitude overlays and shapefile suport for plotting political and tectonic boundaries along the Earth’s surface. In this notebook, we focus on tomographic models of North America and the Western U.S., where high resolution models and rapidly varying seismic properties provide a rich dataset to explore systematically at a range of lengthscales. The notebook demonstrates loading and rendering of IRIS netcdf models, highlighting interesting 3D features of the Western U.S. upper mantle, and goes on to demonstrate how having direct control of the transfer functions used in creating the final volume rendering allows for a more systematic exploration of the role of the visualization method in our interpretation of 3D volumetric data. Finally, we conclude by demonstrating some of the semantically-aware capabilities of yt for analysis purposes, and demonstrate how these tools have cross-disciplinary functionality.

The CF (Climate and Forecast) Conventions are a community-developed metadata standard for storing and describing Earth system science data in the netCDF binary data format. Numerous existing FOSS (Free and Open Source Software) and commercial software tools can explore, analyze, and visualize data that is encoded using the CF Conventions. The CF community holds annual workshops to develop, refine, and review enhancements to the CF Conventions and to manage the CF governance and processes. The EarthCube netCDF-CF project worked with the CF community on the development of extensions to netCDF-CF. Several of these have been accepted into the CF Conventions. Work on these extensions involved broad participation by members of the existing netCDF-CF community as well as members of science domains not traditionally represented in the netCDF-CF community. This presentation will provide an update of recent work and an overview of CF plans and future activities.

Oceanographic research cruises produce abundant data, using a wide range of methods and equipment; very often through large collaborative efforts. These research endeavors span a broad array of disciplines and are critical to investigating the interplay between biological, geological, and chemical processes in the ocean systems over space and time. The advent of genomic sequencing technologies allows for the analysis of gene expression in a variety of environmental settings, to measure the distribution and significance of metabolites and lipids in organisms and the environment. Despite scientists’ best efforts to carefully curate and share their data with collaborators to advance individual studies and publications, no systematic, unifying framework currently exists to integrate ‘omics data with physical, geochemical, and biological datasets commonly used by the broader geoscience community. As a result, the moment each sample leaves the ship is often the last time each data component appears together in a unified collection. Typically, ‘omics datasets are submitted to nucleotide sequence repositories, whereas contextual environmental data are submitted and stored in specialized data-repositories, or only made available within published papers. This makes it difficult to fully reconnect in-situ data, therefore limiting their reuse in other studies. The development of resources to facilitate the aggregation, publication and reuse of biological datasets along with their physicochemical information is critical for studying marine microbes and the biogeochemical processes in the ocean that they drive. We present Planet Microbe, a cyberinfrastructure resource enabling data discovery and open data sharing for historical and on-going oceanographic sequencing efforts. Several historical oceanographic ‘omics datasets (Hawaii Ocean Time-series (HOT), Bermuda Atlantic Time-series (BATS), Global Ocean Sampling Expedition (GOS)) have been integrated into Planet Microbe along with new oceanic large-scale datasets as the Tara Expeditions and Ocean Sampling Day (OSD). In Planet Microbe, these ’omics data have been reintegrated with their in-situ environmental contextual data, including biological and physicochemical measurements, and information about sampling events, and sampling stations. Finally, cruise tracks, protocols and instrumentation are also linked to these datasets to provide the user with a comprehensive view of the metadata. Additionally, Planet Microbe integrates computational tools using National Science Foundation (NSF) funded Cyberinfrastructure (CyVerse) and provides users with free access to large-scale computing power to analyze and explore these datasets.