Robert R. McCormick School of Engineering and Applied Science Electrical Engineering and Computer Science Department Center for Ultra-scale Computing and Information Security at Northwestern University


PnetCDF development was sponsored by the Scientific Data Management Center (SDM) under the DOE program of Scientific Discovery through Advanced Computing (SciDAC). It was also supported in part by National Science Foundation under the SDCI HPC program award numbers OCI-0724599 and HECURA program award numbers CCF-0938000. PnetCDF is currently supported in part by DOE Award Number DE-SC0007456.

Project Team Members:

Northwestern University

Argonne National Laboratory

Northwestern University - EECS Dept.

Parallel netCDF Coverity Scan Build Status

Parallel netCDF (PnetCDF), a collaborative work of Northwestern University and Argonne National Laboratory, is a parallel I/O library for accessing NetCDF files in CDF and CDF-2 formats ¹. In addition, PnetCDF supports the CDF-5 file format, an extension of CDF-2, that allows users to define large dimensions, attributes, variables (> 2B array elements).

¹NetCDF supports parallel I/O starting from version 4, but the file format is restricted to HDF5. PnetCDF is currently the only choice to perform parallel I/O on files in classic formats (CDF-1 and 2). Incorporation of PnetCDF into netCDF is undergoing. See news items below.



What is netCDF?

NetCDF (Network Common Data Form) is an I/O library that supports the creation, access, and sharing of scientific data. Unidata provides implementations of netCDF software. Prior to version 4, netCDF APIs do not support parallel I/O. Although concurrent read from multiple application clients on a shared file can be achieved using individual file pointers, there is no parallel file access semantics in the APIs. For parallel programs, write operations must be done by shipping data to a single process which then writes to the file. Thus, the communication contention on the writing process can make the I/O performance considerably slow.

Starting from version 4, Unidata's netCDF supports parallel I/O either through PnetCDF or HDF5. Through PnetCDF, netCDF-4 can access files in CDF formats in parallel. Similarly, through HDF5, netCDF-4 can access files in HDF5 format.

Design Strategy for Parallelizing NetCDF:

The goal of PnetCDF is to provide high-performance parallel I/O to the applications by enabling all client processes to access a shared file in parallel. To ensure the performance and portability, PnetCDF is built on top of MPI-IO. The PnetCDF APIs incorporate the parallel semantics following the MPI (Message Passing Interfaces) standard and provide backward compatibility with the netCDF file formats: CDF (or CDF-1), CDF-2, and CDF-5. Figure 1 compares the data access from multiple processes between using the sequential netCDF and PnetCDF.

PnetCDF figure comparing sequential I/O, parallel I/O, and parallel I/O with request aggregation feature.
Figure 1. Comparison of data access between using sequential netCDF and Parallel netCDF. (a) Write operation is through one of the clients when using netCDF. (b) PnetCDF enables concurrent write to parallel file systems. (c) Through nonblocking I/O, PnetCDF can aggregate multiple requests into large ones so a better performance can be achieved.

New features added in PnetCDF:

Interoperability with netCDF-4

Download Source Codes

User Documents and Example Programs

Under Development

PnetCDF is constantly updated with new features and performance improvement methods to meet the high-performance computing demands. The following list some task currently under development.

I/O Performance Benchmarking Programs


User Discuss Mailing List

Related Links

User Community:


We are grateful to the following people who provide valuable comments/discussions to improve our implementation.
Yu-Heng Tseng (LBNL) Reiner Vogelsang (Silicon Graphics, Germany), Jon Rhoades (Information Systems & Technology ENSCO, Inc.), Kilburn Building (University Of Manchester), Foucar, James G (Sandia National Lab.), Drake, Richard R (Sandia National Lab.), Eileen Corelli (Senior Scientist, ENSCO Inc.), Roger Ting, Hao Yu, Raimondo Giammanco, John R. Tannahill (Lawrence Livermore Nattional. Lab.), Tyce Mclarty (Lawrence Livermore Nattional. Lab.), Peter Schmitt, Mike Dvorak (LCRC team, MCS ANL)

Northwestern University EECS Home | McCormick Home | Northwestern Home | Calendar: Plan-It Purple
© 2011 Robert R. McCormick School of Engineering and Applied Science, Northwestern University
"Tech": 2145 Sheridan Rd, Tech L359, Evanston IL 60208-3118  |  Phone: (847) 491-5410  |  Fax: (847) 491-4455
"Ford": 2133 Sheridan Rd, Ford Building, Rm 3-320, Evanston, IL 60208  |  Fax: (847) 491-5258
Email Director

Last Updated: $LastChangedDate: 2016-05-03 08:36:25 -0500 (Tue, 03 May 2016) $