| Home | | | Schedule | | | Local Info |
| netCDF/HDF - A Foundation for Sharing NOAA Data |
| Session Time | January 9th; 3:30 pm to 5:00 pm | |||
| Location | White Flint | |||
| Description | NetCDF/HDF is the standard format for important NOAA satellite programs (JPSS and GOES-R), for data exchange in AWIPS, a key element in climate models and results, an international standard in the hydrographic community (BAG) and a standard for all STAR satellite products. It is an important part of the NOAA-wide environmental data foundation. This session has two goals: to improve cross-line office awareness and communication among developers and users of netCDF/HDF and to initiate a discussion with the creators of netCDF/HDF about new capabilities and enhancements that would improve the format and related tools for the NOAA user community. | |||
| Chair | Ajay Krishnan | |||
| Presentations and Notes | Click Here! | |||
| Talk | Length (min) | Title | Presenter |
| 4C.1 | 15 | The HDF Data Format: High Performance Interoperability for Earth Science Communities | Ted Habermann |
| 4C.2 | 15 | Advancing netCDF-CF for the Geoscience Community | Ethan Davis |
| 4C.3 | 15 | NetCDF Gold Standard Examples: A walkthrough of the NCEI templates | Mathew Biddle |
| 4C.4 | 15 | IOOS Compliance Checker | Luke Campbell |
| 4C.5 | 15 | Accessing netCDF Quality -- Tools for catalogs and files. | Roland Schweitzer |
| 4C.6 | 15 | Open Discussion | |
| Abstracts | |||
| 4C.1 | The HDF Data Format: High Performance Interoperability for Earth Science Communities
Ted Habermann (The HDF Group/Earth Science) John Kozimor (The HDF Group), Sean Gordon (The HDF Group) |
||
| Effective interdisciplinary research requires sharing of well-described data in formats that are usable across platforms and languages. Many communities in the earth sciences have turned to HDF5 to provide a stable and flexible format that they can customize with their own community conventions and standards. The HDF format serves these communities especially well because it is a high performance, self-describing format that has implementations and associated tools across many platforms and popular languages. The Common Framework for Earth Observations recommends HDF5 and netCDF4 (an API and data structure built on HDF5) as standard formats for numerical data and associated metadata. We describe several examples of community conventions and standards used to customize the HDF5 format to address specific needs, in order to illustrate the similarities among communities and data types. The intended outcome of this exposure is to foster expertise sharing among communities resulting in increased data and convention interoperability. | |||
| 4C.2 | Advancing netCDF-CF for the Geoscience Community
Ethan Davis (UCAR/Earthcube) |
||
| The Climate and Forecast (CF) metadata convention for netCDF (netCDF-CF) is a community-developed convention
for encoding geoscience data stored in the netCDF binary data format in a self-describing manner. Now an OGC
standard, it can encode information that describes coordinate systems, the geophysical meaning and units of each
variable, and how the data were collected. It can capture this information for a number of scientific feature types (e.g.,
station, sounding, and gridded data). It is widely used by weather forecasters, climate scientists and remote-sensing
researchers. Numerous open source and commercial software tools are able to explore and analyze netCDF-CF datasets.
This presentation will provide an overview and update on work to extend the existing netCDF-CF metadata convention in ways that will broaden the range of earth science domains whose data can be represented. It will include discussion of the enhancements to netCDF-CF that are underway; the current CF community-based standards development process (including validation); and integration with existing and emerging tools. |
|||
| 4C.3 | NetCDF Gold Standard Examples: A walkthrough of the NCEI templates
Mathew Biddle (NESDIS/NCEI/IOOS/CICS) |
||
| The NOAA National Centers for Environmental Information (NCEI) have developed netCDF templates based on what are called "feature types" by Unidata and CF. These templates conform to Unidata's netCDF Attribute Convention for Dataset Discovery (ACDD) and netCDF Climate and Forecast (CF) conventions. Adding to these established conventions, NCEI also provides several recommendations for both netCDF variables and attributes. These best practices capture NCEI's experience in providing long-term preservation, scientific quality control, product development, and multiple data re-use beyond its original intent. In conjunction with the templates, NCEI has provided ‘gold standard’ example files which follow the templates and a collection of reports which document the results of testing the files in various compliance checkers. This presentation will review the NCEI templates and provide information regarding the gold standard examples and subsequent reports. | |||
| 4C.4 | IOOS Compliance Checker
Luke Campbell (Applied Science Associates/IOOS Program Office) |
||
| As the volume of scientific data collection increases the importance of adhering to standard protocols for data
discovery and distribution becomes increasingly important. High-quality metadata is critical to improving searchability
of and access to these scientific datasets. Standardizing metadata allows integration with data catalogs and facilitates
data distribution. There are several community standards that help to improve the quality of metadata and thus prepare
datasets for cataloging. These include the Attribute Convention for Dataset Discovery (ACDD) and Climate and
Forecast Metadata (CF).
To assist data providers with generation of quality metadata, the IOOS Program Office developed the Compliance Checker. The Compliance Checker provides an efficient means for data providers to determine how well a dataset meets various standards and provides guidance for improving the quality of metadata. The tool, written in Python, can be run either from the command line or via a web-based interface. The core checker provides scores for ACDD and CF standards; a series of plug-ins allow data providers to check compliance with other standards such as the NOAA National Center for Environmental Information (NCEI) 2.0 standard. The results of the compliance checker include not only a compliance score, but a report detailing actions that users can take to improve metadata. Use of the Compliance Checker has already directly benefited data providers wanting to submit data to the IOOS Catalog, IOOS Glider Data Assembly Center, or to NCEI for permanent archiving. |
|||
| 4C.5 | Accessing netCDF Quality -- Tools for catalogs and files.
Roland Schweitzer (OAR/Contractor for NOAA/PMEL) Kevin O'Brien (JISAO), Sean Arms (Unidata), Dave Neufeld (NCEI) |
||
| The ncISO Tool traverses THREDDS Catalogs, reads dataset documentation and translates that documentation into different views using Extensible Stylesheet Language Transformations (XSLT) which allow users to assess the quality of the data source with regard to it's coverage of the Attribute Convention for Data Discovery. We have also developed a tool for traversing THREDDS catalogs (the Catalog Evaluator) which also collects information about the underlying netCDF data sources and produces a display of quality information regarding whether the data complies with known conventions and whether it is aggregated appropriately. Recently we complete a project to convert the ncISO to consider the latest ADDC specification (from 1.2 to 1.3) and have incorporated ncISO output in our Catalog Evaluator output. We will report on our experiences updating ncISO, incorporating ncISO output and its implications for data quality across NOAA and beyond. | |||
| Top of the Page |
