ds format

Documentation version: 2.0.1 3.0.0 3.1.0 3.2.0 3.3.3 3.4.0 3.5.2 3.6.1 3.7.0 4.0.1 4.1.1 4.1.2 (latest) master
GitHub

Storage format

ds-format implements a native format “ds” (file extension “.ds”) in addition to other standard formats. This format is compatible with the most commonly used features of NetCDF. Over NetCDF, this format has an advantage of being much simpler and faster (up to 10 times), especially when reading or writing many small files. Unlike NetCDF, which is editable, the ds format is intended to be written only once (the copy-on-write paradigm). This greatly simplifies the format and implementation, and improves performance. The ds format can be used by reading or writing files an extension “.ds”. Existing NetCDF files can be converted to ds with ds select input.nc output.ds, where input.nc is the input NetCDF files and output.ds is the output ds file.

The format is composed of a version string, header and body, separated by a newline character (\n). The version string is ds-<x>.<y>, where x and y are a major and minor format version numbers. The header consists of one line containing JSON of the metadata. In addition to the standard structure, the metadata contains a number of special fields describing where to find and how to decode the variable data. The body is a block of binary data directly following the header. The header and body are separated by a single newline character (\n). The body contains raw binary data of the variables in a sequential order. Schematically, the structure of the format is:

# Three lines of the format version string, header and body.
ds = ds-<x>.<y>\n<header>\n<body>

# JSON header describing variable metadata.
header = { "<var>": <var-metadata> ... ".": <dataset-metadata> }

# Body composed of binary data.
body = <var-data>...

var-data = [<missing-bitmask>]<data>

where var is a variable name, var-metadata is variable metadata, dataset-metadata is dataset metadata, missing-bitmask is a missing value bitmask (described below) and data are binary variable data.

In addition to the standard variable metadata, the ds native format uses the following properties:

Property Description
.offset Data offset in bytes relative to the start of the body.
.len Length of data in bytes, including a missing data bitmask or string lengths, if present.
.type Data type of the variable. One of: float32 and float64 (32-bit and 64-bit floating-point number, resp.), int8, int16, int32 and int64 (8-bit, 16-bit, 32-bit and 64-bit integer, resp.), uint8, uint16, uint32 and uint64 (8-bit, 16-bit, 32-bit and 64-bit unsigned integer, resp.), bool (boolean), str (byte string) and unicode (Unicode).
.endian Endianness. b for big endian, l for little endian.
.missing A boolean value signifying if the data array is a masked array. A bitmask of missing data is stored directly after the variable data, and is bitpacked.

If missing values are allowed (.missing is true), a missing value bitmask is stored at the variable offset. The bitmask is bitpacked, and at the end it is padded with zeros to occupy an integer number of bytes. Bit ordering of bitpacked values is always big endian, regardless of .endian of the variable.

The variable data are stored directly after missing value bitmask, or at variable offset if missing values are not allowed. They are stored as a flat sequence of binary values in bit ordering as in .endian. Multi-dimensional arrays are stored in the “C ordering” of rows and columns. Missing values are not written.

Boolean values (type bool) are bitpacked, and at the end are padded with zeros to occupy an integer number of bytes. Bit ordering of bitpacked values is always big endian, and .endian of boolean variables should be b.

Data of string arrays (type str and unicode) are stored as an array of string lengths, followed by a sequence of the strings, encoded as UTF-8 if the original strings are Unicode. The array of lengths is a flat array of 64-bit unsigned integers in bit ordering as in .endian. The strings are stored directly following this array as a sequence of bytes, with no separators between the strings.

Performance

The ds format is up 10 times faster than NetCDF, while taking the same or less space (uncompressed). It is especially faster for reading and writing small files. Below are results of a set of performance tests which write and read NetCDF and ds files:

Test Time NetCDF (s) Time ds (s) Speed factor Size NetCDF (MB) Size ds (MB) Size factor
write tiny 100k 56 11 5 394 394 1
write small 100k 82 12 7 1566 785 2
write large 10 11 11 1 7633 7633 1
read tiny 100k 60 6 10      
read small 100k 70 8 9      
read large 10 3.3 2.5 1.3      

The factors are NetCDF relative to ds.