Appendix G CDF-5 file format specification

Notes

Note on vsize:This number is the product of the dimension lengths (omitting the record dimension) and the number of bytes per value (determined from the type), increased to the next multiple of 4, for each variable. If a record variable, this is the amount of space per record (except that, for backward compatibility, it always includes padding to the next multiple of 4 bytes, even in the exceptional case noted below under "Note on padding"). The netCDF "record size" is calculated as the sum of the vsize’s of all the record variables.

The vsize field is actually redundant, because its value may be computed from other information in the header. The 32-bit vsize field is not large enough to contain the size of variables that require more than 2^32 - 4 bytes, so 2^32 - 1 is used in the vsize field for such variables.

Note on names:CDF-1 enforces a more restricted set of characters in creating new names, but permits reading names containing arbitrary bytes. This specification extends the permitted characters in names to include multi-byte UTF-8 encoded Unicode and additional printing characters from the US-ASCII alphabet. The first character of a name must be alphanumeric, a multi-byte UTF-8 character, or ’_’ (reserved for special names with meaning to implementations, such as the "_FillValue" attribute). Subsequent characters may also include printing special characters, except for ’/’ which is not allowed in names. Names that have trailing space characters are also not permitted.

Implementations of all CDF formats must ensure that names are normalized according to Unicode NFC normalization rules during encoding as UTF-8 for storing in the file header. This is necessary to ensure that gratuitous differences in the representation of Unicode names do not cause anomalies in comparing files and querying data objects by name.

Note on streaming data:The largest possible record count, 2^32 - 1, is reserved to indicate an indeterminate number of records. This means that the number of records in the file must be determined by other means, such as reading them or computing the current number of records from the file length and other information in the header. It also means that the numrecs field in the header will not be updated as records are added to the file. [This feature is not yet implemented].

Note on padding:In the special case when there is only one record variable and it is of type character, byte, or short, no padding is used between record slabs, so records after the first record do not necessarily start on four-byte boundaries. However, as noted above under "Note on vsize, the vsize field is computed to include padding to the next multiple of 4 bytes. In this case, readers should ignore vsize and assume no padding. Writers should store vsize as if padding were included.

Note on byte data:It is possible to interpret byte data as either signed (-128 to 127) or unsigned (0 to 255). When reading byte data through an interface that converts it into another numeric type, the default interpretation is signed. There are various attribute conventions for specifying whether bytes represent signed or unsigned data, but no standard convention has been established. The variable attribute "_Unsigned" is reserved for this purpose in future implementations.

Note on char data:Although the characters used in netCDF names must be encoded as UTF-8, character data may use other encodings. The variable attribute "_Encoding" is reserved for this purpose in future implementations.

Note on fill values: Because data variables may be created before their values are written, and because values need not be written sequentially in a netCDF file, default "fill values" are defined for each type, for initializing data values before they are explicitly written. This makes it possible to detect reading values that were never written. The variable attribute "_FillValue", if present, overrides the default fill value for a variable. If _FillValue is defined then it should be scalar and of the same type as the variable.

Fill values are not required, however, because netCDF libraries have traditionally supported a "no fill" mode when writing, omitting the initialization of variable values with fill values. This makes the creation of large files faster, but also eliminates the possibility of detecting the inadvertent reading of values that haven’t been written.

Notes on Computing File Offsets

The offset (position within the file) of a specified data value in a CDF-1, CDF-2, or CDF-5 file is determined by the variable start location (the offset in the begin field), the external type of the variable (the nc_type field), and the dimension indices (one for each of the variable’s dimensions) of the value desired.

The external size in bytes of one data value for each possible netCDF type, denoted extsize below, is:

The record size, denoted by recsize below, is the sum of the vsize fields of record variables (variables that use the unlimited dimension), using the actual value determined by dimension sizes and variable type in case the vsize field is too small for the variable size.

To compute the offset of a value relative to the beginning of a variable, it is helpful to precompute a “product vector” from the dimension lengths. Form the products of the dimension lengths for the variable from right to left, skipping the leftmost (record) dimension for record variables, and storing the results as the product vector for each variable.

For example:

Non-record variable:

        dimension lengths:      [  5  3  2 7]
        product vector:         [210 42 14 7]

Record variable:

        dimension lengths:      [0  2  9 4]
        product vector:         [0 72 36 4]

At this point, the leftmost product, when rounded up to the next multiple of 4, is the variable size, vsize, in the grammar above. For example, in the non-record variable above, the value of the vsize field is 212 (210 rounded up to a multiple of 4). For the record variable, the value of vsize is just 72, since this is already a multiple of 4.

Let coord be the array of coordinates (dimension indices, zero-based) of the desired data value. Then the offset of the value from the beginning of the file is just the file offset of the first data value of the desired variable (its begin field) added to the inner product of the coord and product vectors times the size, in bytes, of each datum for the variable. Finally, if the variable is a record variable, the product of the record number, ’coord[0]’, and the record size, recsize, is added to yield the final offset value.

A special case: Where there is exactly one record variable, we drop the requirement that each record be four-byte aligned, so in this case there is no record padding.

Examples

By using the grammar above, we can derive the smallest valid netCDF file, having no dimensions, no variables, no attributes, and hence, no data. A CDL representation of the empty netCDF file is

netcdf empty { }

This empty netCDF file has 32 bytes if it is in CDF-1 or CDF-2 format. If the file is in CDF-5 formatm the size is 48 bytes. For CDF-1 and CDF-2 format, the file begins with the four-byte “magic number” that identifies it as a netCDF version 1 file: ‘C’, ‘D’, ‘F’, ‘\x01’. Following are seven 32-bit integer zeros representing the number of records, an empty list of dimensions, an empty list of global attributes, and an empty list of variables. The layout is similar for CDF-5, but the NON_NEG and ZERO64 integers are 64-bits.

Let’s use an example file named tiny.nc created from a CDL file shown below.

% cat tiny.cdl
netcdf tiny {
dimensions:
        dim = 5;
variables:
        short vx(dim);
data:
        vx = 3, 1, 4, 1, 5 ;
}

% ncgen -v 5 -b tiny.cdl

% ncgen -v 2 -b tiny.cdl

Below is an (edited) dump of tiny.nc produced using the Unix command

% od -c tiny.nc

0000000    C   D   F 005  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \n
          [ magic number] [    numrecs = 0 records      ] [NC_DIMENSION ]
0000020   \0  \0  \0  \0  \0  \0  \0 001  \0  \0  \0  \0  \0  \0  \0 003
          [     nelems = 1 dimension    ] [  dimension name has 3 chars  ]
0000040    d   i   m  \0  \0  \0  \0  \0  \0  \0  \0 005  \0  \0  \0  \0
          [ name="dim"  ] [       dim_length = 5        ] [ 0 global att-
0000060   \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \v  \0  \0  \0  \0
          ribute (ABSENT = ZERO ZERO64) ] [ NC_VARIABLE ] [  nelems =
0000100   \0  \0  \0 001  \0  \0  \0  \0  \0  \0  \0 002   v   x  \0  \0
           1 variable   ] [ variable name has 2 chars   ] [ name = "vx" ]
0000120   \0  \0  \0  \0  \0  \0  \0 001  \0  \0  \0  \0  \0  \0  \0  \0
          [ variable has 1 dimension    ] [  first dimension has ID 0   ]
0000140   \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 003
          [ var has 0 attributes (ABSENT = ZERO ZERO64) ] [type=NC_SHORT]
0000160   \0  \0  \0  \0  \0  \0  \0  \f  \0  \0  \0  \0  \0  \0  \0 200
          [  var size, vsize = 12 bytes ] [var's file offset begin = 128]
0000200   \0 003  \0 001  \0 004  \0 001  \0 005 200 001                
          [    3] [    1] [    4] [    1] [    5][padding]
0000214

0000000    C   D   F 002  \0  \0  \0  \0  \0  \0  \0  \n  \0  \0  \0 001
          [ magic number] [ numrecs = 0 ] [NC_DIMENSION ] [ 1 dimension ]
0000020   \0  \0  \0 003   d   i   m  \0  \0  \0  \0 005  \0  \0  \0  \0
          [  3 chars    ] [ name="dim"  ] [dim_length=5 ] [ 0 global atts
0000040   \0  \0  \0  \0  \0  \0  \0  \v  \0  \0  \0 001  \0  \0  \0 002
          (ABSENT)      ] [ NC_VARIABLE ] [  1 variable ] [   2 chars   ]
0000060    v   x  \0  \0  \0  \0  \0 001  \0  \0  \0  \0  \0  \0  \0  \0
          [ name = "vx" ] [ 1 dimension ] [ 1st dim ID 0] [ 0 attributes
0000100   \0  \0  \0  \0  \0  \0  \0 003  \0  \0  \0  \f  \0  \0  \0  \0
           (ABSENT)     ] [type=NC_SHORT] [vsize=12bytes] [ var's file offset
0000120   \0  \0  \0   T  \0 003  \0 001  \0 004  \0 001  \0 005 200 001
           begin = 84   ] [    3] [    1] [    4] [    1] [    5][padding]
0000140

Note that if the netCDF files were generated by utility "ncmpigen" instead of "ncgen", the file size and the value of variable’s begin will be different. This is because PnetCDF uses a default minimum file header size of 512 bytes. Thus the last three lines of the dump will become:

0000120   \0  \0 002  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
          begin = 512   ] [    skip ....
0000140   \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
*
0001000   \0 003  \0 001  \0 004  \0 001  \0 005
         ][    3] [    1] [    4] [    1] [    5]
0001012