Sample data storage format

Options

The format to expect for all schedules can be determined by the storage command (see Related commands below) or from the metadata entry data_format in the Deployment section (section 4) metadata.  Once the value for normal deployments is known for a given instrument, it can be assumed that it will not change during the instrument's lifetime.  The only case when the format may be different is if the deployment was enabled in calibration mode, in which case the format will be calfloat64.

Layout

Within the sample data for a given schedule, a single sample with N channels (N > 0) will have the form:

All individual data items are stored in memory in little-endian format (least significant byte at lower address).  Within a given schedule, all samples occupy the same number of bytes each.

The Timestamp is a 64-bit count of milliseconds elapsed since 1970/01/01 00:00:00, commonly referred to as the Unix epoch.  It accounts for leap years, but not leap seconds, time zones or any other adjustments.  Note that although the Unix epoch is used as a reference point, the time does not conform exactly to the definition of "Unix time", which is measured in seconds.

All channel values have the same numeric format; one of float32, float64 or calfloat64 (see Options, above).  The order in which channel values appear is determined by:

1. The order in which groups are specified for this schedule.

2. Within each group, the order in which channels are specified.

These orderings are determined by the schedule and group commands (see Related commands, below).

All channel values have the potential to encode an error value.  Error values are denoted by NaNs with an embedded payload (see Errors, below)

Related commands

instrument

>> instrument datatype
<< instrument datatype=float32 | float64 | calfloat64

• Read-only

• Sample data is in the same format for all channels.

• Normal deployments will be in either float32 or float64, the choice is configured at the Factory.

enable

>> enable config=<config_label> dataset=<dataset_label> storagemode=normal | calibration
<< enable config=<config_label> dataset=<dataset_label> storagemode=normal | calibration

• storagemode is optional and defaults to normal if not specified.

• normal selects a data storage format of either float32 or float64, according to the logger's configuration.

• calibration always selects a data storage format of calfloat64, regardless of the normal setting.

schedule

>> schedule grouplist=<group_label_list ...>
<< schedule grouplist=<group_label_list ...>

• The order in which group labels are specified determines the order in which groups appear in the stored data.

group

>> group channellist=<channel_label_list ...>
<< group channellist=<channel_label_list ...>

• The order in which channel labels are specified determines the order in which channels appear within their group in the stored data.

dataset

>> dataset <dataset_label> datatype
<< dataset <dataset_label> datatype=float32 | float64 | calfloat64

• The data storage format of a historical dataset held in the logger's memory can be queried at any time.

Errors

Errors related to channels are denoted by negative-signed NaN values.  In many cases, detecting a NaN as a channel value should suffice to indicate an error has occurred since most error codes are diagnostic/correctable for RBR-only.  For conversion from float32 to float64, a 29-bit least-significant-bit-zero-pad of the payload is used (mantissa is left-shifted 29 bits).  Conversely, for float64 to float32, a 29-bit least-significant-bit-truncation of the payload is used (mantissa is right-shifted 29 bits).  This conversion matches several C compiler and Java intrinsic conversions, but integrators should check intrinsic conversions in their own environments.  Except for the case of a general error, the setting of the quiet/signalling bit (float32: 0x00400000, float64: 0x0008000000000000) should not be relied on for error determination; for a general error, the bit will be set.