Next: Nonblocking Read, Previous: Adding, Up: Use of the PnetCDF Library [Index]
PnetCDF nonblocking APIs allow users to first post multiple requests and later flush them altogether in order to achieve a better performance. A common practice is writing (or reading) subarrays to (from) multiple variables, e.g. one or more subarrays for each variable defined in the NetCDF file. Without the need of immediate commitment of each request, nonblocking APIs allows PnetCDF to aggregate the posted requests (often noncontiguous and relatively of small size) into a contiguous large one, which is handled more efficiently by the underneath MPI-IO library and file systems. Programming of nonblocking APIs consists of two parts: posting the nonblocking requests and later flushing the pending requests. APIs ncmpi_wait_all and ncmpi_wait flush the pending requests by aggregating them into a single MPI-IO call. Here is a typical sequence of PnetCDF API calls to use nonblocking APIs to write to a netCDF file:
ncmpi_create /* create netCDF file and enter define mode */
...
ncmpi_enddef /* end definitions: leave define mode */
...
ncmpi_iput_var<kind>_<type> /* post a nonblocking write request to a variable */
...
ncmpi_iput_var<kind>_<type> /* post another nonblocking write request to the same or a different variable */
...
ncmpi_wait_all /* commit the posted nonblocking requests */
...
ncmpi_buffer_attach /* tell PnetCDF the amount of space for write request cache */
...
ncmpi_bput_var<kind>_<type> /* post a buffered nonblocking write request to a variable */
...
ncmpi_bput_var<kind>_<type> /* post another buffered nonblocking write request to a variable */
...
ncmpi_wait_all /* commit the posted nonblocking requests */
...
ncmpi_buffer_detach /* tell PnetCDF to free the write cache */
...
ncmpi_close /* close: save new netCDF file */
Nonblocking APIs are not collective. They can be called in either collective or independent data mode. Starting from 1.7.0, the nonblocking APIs can also be called in define mode. Multiple nonblocking I/O requests can be posted and the number of nonblocking requests can be different among MPI processes.
There are two wait APIs for committing the pending nonblocking requests: ncmpi_wait_all() to be called in the collective data mode and ncmpi_wait() in the independent data mode.
For write operations, there are also two kinds of nonblocking APIs, one with the name "iput" and the other "bput". For "iput" APIs, users should not alter the contents of the write buffer once the request is posted until the wait API is returned. Any change to the buffer contents in between will result in unexpected error.
The write buffer used in one "iput" call should not be reused in another "put/get" API call (blocking or nonblocking) and buffers used in different "iput/put/iget/get" API calls should not overlap. Currently, PnetCDF does not detect whether user buffers are reused or overlapped.
To alleviate the above limitation, the "bput" API family can be used to save a copy of the request into an internal memory cache. Thus, once a bput nonblocking request is posted, users are free to change the contents of the write buffer. Before using "bput" APIs, users are required to tell PnetCDF the amount of space (the high water mark, not total) allowed to cache the write requests. The two APIs that tell PnetCDF to allocate the cache amount and to free the space are ncmpi_buffer_attach() and ncmpi_buffer_detach(), respectively.
Using ncmpi_iput_var<kind>:
int req[NUM_VARS], statuses[NUM_VARS];
/* write to one variable at a time using iput */
for (i=0; i<NUM_VARS; i++) {
err = ncmpi_iput_vara_int(ncid, varids[i], starts, counts, buf[i], &req[i]);
handle_error(err);
}
/* Note the data contents in write buffer buf[i] should not be touched */
/* wait for the nonblocking writes to complete */
err = ncmpi_wait_all(ncid, NUM_VARS, req, statuses);
if (err != NC_NOERR) {
handle_error(err);
/* check errors for individual requests */
for (i=0; i<NUM_VARS; i++)
handle_error(statuses[i]);
}
/* Only after ncmpi_wait_all returns, the contents in buf[i] can be changed */
Using ncmpi_bput_var<kind>:
int req[NUM_VARS], statuses[NUM_VARS]; /* attach an internal buffer of size equal to the sum of all requests */ err = ncmpi_buffer_attach(ncid, SUM_OF_ALL_REQUESTS); if (err != NC_NOERR) handle_error(err); /* write to one variable at a time using bput */ for (i=0; i<NUM_VARS; i++) { err = ncmpi_bput_vara_int(ncid, varids[i], starts, counts, buf[i], &req[i]); handle_error(err); } /* Note the data contents of write buffer buf[i] can be freely changed, because * they have been copied to a PnetCDF internal memory buffer of size set in the * call to ncmpi_buffer_attach(), Any content change to buf[i] after bput call * returns will not appear in the file..*/ /* wait for the nonblocking writes to complete */ err = ncmpi_wait_all(ncid, NUM_VARS, req, statuses); if (err != NC_NOERR) { handle_error(err); /* check errors for individual requests */ for (i=0; i<NUM_VARS; i++) handle_error(statuses[i]); } /* release the internal buffer */ err = ncmpi_buffer_detach(ncid); handle_error(err);
Next: Nonblocking Read, Previous: Adding, Up: Use of the PnetCDF Library [Index]