Robert R. McCormick School of Engineering and Applied Science Electrical Engineering and Computer Science Department Center for Ultra-scale Computing and Information Security at Northwestern University

Sponsor:

PnetCDF development was sponsored by the Scientific Data Management Center (SDM) under the DOE program of Scientific Discovery through Advanced Computing (SciDAC). It was also supported in part by National Science Foundation under the SDCI HPC program award numbers OCI-0724599 and HECURA program award numbers CCF-0938000. PnetCDF is currently supported in part by DOE Award Number DE-SC0007456.

Project Team Members:

Northwestern University

Argonne National Laboratory






Northwestern University - EECS Dept.



Parallel netCDF


Parallel netCDF (PnetCDF) is an I/O library that supports data access to netCDF files in parallel, a collaborative work of Northwestern University and Argonne National Laboratory.

News:

About netCDF:

NetCDF (Network Common Data Form) defines two sets of standards to support the creation, access, and sharing of scientific data. Unidata provides implementations of netCDF. Prior to version 4, netCDF APIs do not support parallel I/O. Although concurrent read from multiple application clients on a shared file can be achieved using individual file pointers, there is no parallel file access semantics in the APIs. For parallel write operations, data must be shipped to a single process and the written to a file by that process. Hence, the communication contention on the writing process can make the parallel I/O performance considerably slow.

Starting from version 4, Unidata's netCDF supports parallel I/O either through PnetCDF or HDF5. Through PnetCDF, netCDF-4 can access files in CDF formats in parallel. Similarly, through HDF5, netCDF-4 can access files in HDF5 format.

Figure 1. Comparison of data access between using sequential netCDF and Parallel netCDF. (a) Write operation is through one of the clients when using netCDF. (b) PnetCDF enables concurrent write to parallel file systems. (c) Through nonblocking I/O, PnetCDF can aggregate multiple client requests into large ones to achieve better performance.

Parallel netCDF features:

PnetCDF contains a set of new APIs for accessing netCDF files in parallel. Figure 1 compares the data access from multiple processes between using the sequential netCDF and PnetCDF. The new APIs incorporate the parallel semantics following the Message Passing Interfaces (MPI) and provide backward compatibility with the netCDF file formats: CDF (or CDF-1), CDF-2, and CDF-5. Other features of PnetCDF are:

Interoperability with netCDF-4

Download Source Codes

User Documents

Tutorial

Under Development

PnetCDF is constantly updated with new features and performance improvement methods to meet the high-performance computing demands. The following list some task currently under development.

I/O Performance Benchmarking Programs

Publications

User Discuss Mailing List

Related Links

Users:

Acknowledgements:

We are grateful to the following people who provide valuable comments/discussions to improve our implementation.
Yu-Heng Tseng (LBNL) Reiner Vogelsang (Silicon Graphics, Germany), Jon Rhoades (Information Systems & Technology ENSCO, Inc.), Kilburn Building (University Of Manchester), Foucar, James G (Sandia National Lab.), Drake, Richard R (Sandia National Lab.), Eileen Corelli (Senior Scientist, ENSCO Inc.), Roger Ting, Hao Yu, Raimondo Giammanco, John R. Tannahill (Lawrence Livermore Nattional. Lab.), Tyce Mclarty (Lawrence Livermore Nattional. Lab.), Peter Schmitt, Mike Dvorak (LCRC team, MCS ANL)


Northwestern University EECS Home | McCormick Home | Northwestern Home | Calendar: Plan-It Purple
© 2011 Robert R. McCormick School of Engineering and Applied Science, Northwestern University
"Tech": 2145 Sheridan Rd, Tech L359, Evanston IL 60208-3118  |  Phone: (847) 491-5410  |  Fax: (847) 491-4455
"Ford": 2133 Sheridan Rd, Ford Building, Rm 3-320, Evanston, IL 60208  |  Fax: (847) 491-5258
Email Director

Last Updated: $LastChangedDate: 2014-07-11 18:11:59 -0500 (Fri, 11 Jul 2014) $