The Data Access Protocol Version 4 (DAP4) Specification

Unlike FTP and other protocols enabling clients to access whole files or granules, DAP offers sub-granular retrievals, and this requires exposing the detailed structure of each granule (called a “dataset” in DAP). To this end, the structure of every DAP-accessible dataset is manifest in a machine-readable document called the DMR. DMRs are declaration statements that adhere formally to the DAP Data Model, which is sufficiently general to make retrievable a variety of data granules and data types whose semantics vary widely among the (potential) sources.

The DAP data model is built around a notion of “variables” that fall into three classes—atoms, structures and sequences—and may be organized as multidimensional arrays. As reflected in its DMR, all of a dataset’s variables are named, and they may optionally be grouped (in a hierarchy) to add meaning and allow complex name-driven retrievals by clients. DAP variables are (strongly) typed and optionally may have “attributes” and “dimensions”, the former to clarify meaning and the latter to indicate a variable’s array shape (where relevant). Attributes resemble variables except that the former may be assigned to the latter, but not vice versa.

Much of DAP’s power arises from its typing structure. The value(s) of an atomic variable (whether or not dimensioned as an array) are all of the same atomic type, indicated in the DMR to be integer, floating point, character, string, etc. Structure variables are combinations of atom variables joined for semantic reasons, such as to represent complex numbers, vector-valued parameters, or other relationships among the values of a dataset.

To address a difficult abstraction challenge—retrievals from both array-oriented and database-oriented sources—DAP allows DMRs to declare variables of type “Sequence”. A sequence is like a relation, i.e., a table whose rows are “instances” or “records” and whose columns contain the values of “fields”. In DAP, the content of a sequence may comprise an arbitrary number of instances (i.e., records) whose “fields” are values from other DAP variables.

For example, a sequence named “BirdTracking” might contain variables named “BirdID”, “BandingEvent”, and “ObservedPosition”, and the contents of BirdTracking would be instances that contain values of the specified variables (fields). DAP allows arbitrary nesting of sequences and structures, so in this example BandingEvent could be a structure (containing date and location info for the bird’s banding) and ObservedPosition could be a sequence (containing equivalent date and location info) whose instances represent subsequent bird observations.

Hypothetically, this is akin to making BirdTracking a structure with a single dimension (though DAP does not allow arrays of indefinite length). However, selective retrieval from an array would require knowing the indices of desired instances, whereas a BirdTracking sequence allows filtering-style retrievals that select instances on the basis of their values.

Much about DAP may be discerned from a dictionary or glossary-like list of the key entities in its data model. Such a list follows, sequenced for ease of understanding (rather than alphabetically). These descriptions are not definitive, as the formal specification documents take precedence over anything stated here.

Data Source - Though formally outside the DAP Data Model, the term Data Source generally refers to all the datasets (see below) that may be retrieved via DAP from a single server, identified by its domain name. Servers sometimes offer (catalogs and/or inventories of) collections and sub-collections, but the DAP Data Model focuses on granular and sub-granular retrievals.

Dataset - Sometimes called a “granule” in catalog/inventory parlance, a DAP Dataset is represented by a unique (unadorned) URL, and is the highest-level entity (metadata as well as content) described by the DAP Data Model. Clients invoke retrieval operations by adorning the Dataset URL with suffixes and query strings interpreted by the server.

Declarations and the DMR - For a specific Dataset, all aspects of the DAP Data Model (name assignments, structural definitions, etc) are governed by a formal declarations document. Created as part of making a Dataset DAP-retrievable, this document is dubbed the DMR (roughly: Dataset Metadata Response), and clients may retrieve DMRs alongside or independently of Dataset contents.

Name - Most entities in the DAP Data Model may or must be named. With some constraints on the use of special characters (such as “.”), a Name can be any character string. To avoid conflicts, DAP has scoping rules. For example, a Variable and a Dimension may have the same Name without ambiguity, but two Variables can have the same Name only if they are declared in different Groups or structures.

Variable - The building blocks for the DAP Data Model are Variables, which are strictly typed. Three classes of them (atoms, structures and sequences) are described separately below, but in actuality these are best distinguished by inspecting their Type declarations. Each Variable must be assigned a Type and a Name, and it may optionally have a number of Dimensions and Attributes, elaborated below.

Type - Underpinning DAP’s “container” Types (Structure and Sequence, implied above) are “atomic” Types akin to those of computer languages: bytes, integers, floating-point values, strings, and URLs, plus an enum type (permitting specified character strings, such as days of the week, to be treated as variable values) and an opaque type (permitting arbitrary blobs of bits). More detailed Type descriptions are provided in Volume I of the DAP specification.

(Atom) Variable - A Variable whose Type is atomic (see above) comprises a single value or an array of values, and all its values are of the designated Type. A Variable is an array only if its declaration includes Dimensions, which determine the array’s shape and its element-ordering (see below).

(Structure) Variable - A Variable of Type “Structure” is a container for other variables, often implying relationships among them. For example, a structure Variable named “Velocity” might contain a pair of atom Variables (or fields) named “x” and “y,” representing components of a velocity vector. These components would be retrieved via their “qualified” Names, “Velocity.x” and “Velocity.y”.

Notes on Structures:

(Sequence) Variable - A Variable of Type “Sequence” is a container holding multiple (unordered) instances of other DAP Variables. For example, a sequence Variable named “TracerParticle” might contain a pair of structures named “Velocity” and “Displacement”, each declared—as in an earlier example—to have x and y components. The instances of TracerParticle would be like a set of tabular records whose four fields, Displacement.x, Displacement.y, Velocity.x, and Velocity.y are retrieved via filter-style (rather than indexed) retrievals, as discussed in a later section on Constraints.

Notes on Sequences:

Group - The DAP Data Model has a hierarchical mechanism for grouping Variables and carving out independent namespaces. Groups may be nested, and all but one must have Names, the exception being the root of the hierarchy, where the Dataset itself is a Group (needing no name). Retrieving a Variable whose declaration falls within a Named Group requires use of its fully qualified name (FQN), such as GroupA.Group2.Velocity. Any Group (including the Dataset) may be assigned Attributes but not Dimensions.

Attribute - Otherwise nearly indistinguishable from a Variable, an Attribute must always be assigned to a specific Variable or Group. The purpose of Attributes is to provide context or add meaning to the assigned entities, whereas the purpose of Variables is to convey primary content. Retrieving an Attribute always requires prepending the name of the Variable or Group to which it is assigned, which implies that Attribute Names (such as “Units”) enjoy unlimited reusability.

Dimension - A Dimension must have a size and may have a Name. A Variable of any type may optionally be assigned a number of Dimensions, in which case its (compound) values are organized and retrieved as an indexible array of rank n, where n is the number of assigned Dimensions.

Notes on Dimensions:

Shared Dimensions that serve to indicate relations between different arrays which can be used to build/represent Coverages…

Note: Though adoption to-date has been most pronounced in Earth sciences, DAP’s data types and structures (with the possible exception of coverages, discussed in this section) are not at all specific to these disciplines, so we think DAP is positioned for effective use in many domains, scientific and otherwise.

A client’s first step in selectively retrieving a data source often is to discern the character (i.e., its schema) by requesting what DAP calls the DMR (the data-source metadata response). A DMR provides a complete characterization of the associated data source sans content, spelling out its groups, variables, types, dimensions, and attributes as discussed in the preceding two subsections. For ease of use in client software, the DMR adheres to a formal syntax and most often is delivered as an XML document, though other forms are anticipated as DAP4 extensions.

Though it is common to retrieve its DMR prior to requesting content from a data source, this is not the only option. Indeed, a “Data Request” under DAP returns both the DMR and the content (i.e., the values of variables) for the designated data source, because the former is critical for interpreting the latter.

Under DAP, the requests clients make of servers, and the resulting server responses, are all governed by the protocol specification. As stated previously, the formal specification takes precedent over anything stated here.

For each data source, a number of responses may elicited by a client, determined by adding a suffix and/or a query string to the basic URL for the desired data source. Passing the server a completely unadorned URL yields a Dataset Services Response (DSR). This XML document describes the various DAP services available for that source, and these always include provision of a DMR and provision of content from the source. Unlike the DMR, which is always textual, content (delivered in response to a Data Request, as discussed above) may be conveyed in textual or binary form, the latter minimizing data-transfer volumes, of course.

If the URL for a Data Request includes a query string, the server parses this string to determine what data processing the server should perform before constructing its Data Response. Though other classes of pre-retrieval processing are anticipated to be defined via DAP extensions, two forms are mandated by DAP4 for all servers, Index Subsetting and Field Subsetting, and a third form, Filtering, is defined in the core DAP specification, though its implementation by servers is optional.

Index Subsetting - Choosing parts of an array based on the indexes of that array’s dimensions. This operation always returns an array of the same rank as the original, although the size of the return array will (likely) be smaller. Index subsetting uses the bracket syntax described later.

Field Subsetting - Choosing specific variables or fields from the dataset. A dataset in DAP4 is made up of a number of variables and those may be Structures or Sequences that contain fields (and, in effect, the Dataset is itself a Structure and all of its variables are fields - the distinction is more convenience than formal). Field subsetting using the brace syntax described later. One or more fields can be specified using a semicolon (;) as the separator.

Filtering - A filter is a predicate that can be used to choose data elements based on their values. the vertical bar (|) is used as a prefix operator for the filter predicate. Filters can be applied to elements of an Array or fields of a Sequence. A filter predicate consists of one or more filter subexpressions. One or more subexpressions can be specified, using a comma (,) as the separator.

Other services listed in the DSR might (at the server’s option) include the DAP Asynchronous Response. Where implemented (such as for near-line data sources), this response is sent to the client when the requested resource (DMR, Data Response, etc.) is not immediately available. If, in turn, the client makes a “retrieve it” request, the server will respond with a second Asynchronous Response informing the client about when and where the requested resource may be retrieved.

In addition to the most common data objects, a DAP server may provide additional “services,” such as HTML-formatted representations of a data source’s structure and content. Such additional services are discussed in Volume 2 of the specification.

Summary: DAP4 now supports Groups, a generalized form of a grid datatype, adds a few new atomic types, but Grids are removed from the DAP4 Data model.

The DAP4 data model is fundamentally similar to that for DAP2. New atomic types include: enumeration, 64-bit integer, and opaque, the container types removes Grids but now include Groups (see Figure 1). In DAP4, a DAP2 Grid is represented by an Array together with information about its Dimensions which may be shared, and Maps (see Coverage Variables and Maps). Groups provide a way to organize collections of variables and dimensions and to encode these organizational relationships when they are present in the underlying source data.

Dimensions may now be named, and the presence of shared dimensions (i.e., several variables employ a dimension with a given name) along with explicitly name ‘maps’ serves to indicate relationships among arrays that can, in turn, be used to build/represent a more general form of the DAP2 Grid datatype that resembles the OGC/ISO “discrete coverage” datatype. These ‘discrete coverages’ subsume the role of DAP2 Grids, so the latter have been removed from DAP4.

Migrating from DAP2 to DAP4

For servers: A DAP2 DDS/DAS (or DDX) is very close to a DAP4 DMR (indeed, our C++ library contains a way to build a DMR from a DDS). The set of datatypes supported by DAP4 is almost a proper superset of those in DAP2 (see Figure 1]), the exception being that DAP2’s Grid type has been removed. To represent a DAP2 Grid in DAP4, the components of the DAP2 Grid are retained and the appropriate Shared Dimension and Map elements are added to the dataset/group and array. Since the DAP4 ‘discrete coverage’ type subsumes the DAP2 Grid, it will always be possible to translate a DAP2 Grid into DAP4.

For clients: Some of the new data types are more challenging to implement than the types included with DAP2. Of particular note are Enumerations and the expanded grid (aka ‘discrete coverage’) types.

Summary or the main changes between DAP2 and DAP4 Responses:

In DAP4 there is a single XML document, the DMR (see Figure 1), that encodes the metadata for a data source. This response is conceptually similar to, and in some ways identical too, the DDX response that is supported by many DAP2 servers, so it’s organization will be familiar to many people already. As with DAP2, there is one data response that can be modified (constrained) using a expression to limit the information it includes. The basic concepts of slicing an array are unchanged in DAP4. We’ve taken care to allow servers to extend the information passed into the data retrieval web service, a topic that is covered in a bit more detail below under web services. We have replaced the selection part of the DAP2 constraint expression with a filter sub-expression that is applied to specific variables. This enables two or more Sequences to have their own filtering operations (before that was not possible). Our expanded constraint language also provides a way to subset coverages, and a proposed extension to the filtering sub-expression provides a way to subset arrays/coverages by value.

We wanted DAP4 to fully embrace REST. DAP2, even though it predates the term, including many, but not all, of the REST architecture’s features. One change from DAP2 was to explicitly define what happens when a client dereferences a ‘bare URL’ (one without an extension used to ask for a specific DAP4 response). When a DAP4 sever is asked to return information at a bare URL, the result is a Dataset Services Response (DSR) which contains links to all of the other responses for that dataset. In addition, the DSR may contain other information such as server operations that can be used with the dataset. The DSR is an XML document but can contain a stylesheet that transforms it to HTML for a web browser.

DAP4 servers can also support asynchronous access to data, which enables access to data from near-line devices and can be used for some server processing operations (e.g., operations that take a long time to perform). Asynchronous responses are responses that contain a URL that can be used to retrieve the actual data at some time in the future. The protocol has been designed to reduce the chance that a client will mistakenly make a large number of asynchronous requests since this could present an undue burden on some kinds of near-line devices.

Migrating from DAP2 to DAP4

Summary:

We have added three changes to the encoding of returned data values. All top-level variables in a data response now include a CRC32 checksum of their values. This enables people to see if a request is returning the same data values as it did previously. The checksum values are encoded in Attributes bound to the returned variables. We have added an encoding scheme for data values that preserves compactness yet allows clients to easily detect when a server has encountered an error while sending a response. Similarly, we have adopted a Reader Make Right encoding scheme instead of the network byte order scheme used by DAP2. The latter has become more and more important as the predominance of little-endian processors has increased.

Migrating from DAP2 to DAP4

In many ways the encoding scheme is simpler for servers because the data response uses the server’s native byte order. Clients must detect the byte order and twiddle bytes as needed. However, the server must correctly implement the chunking protocol used by the data response and must correctly computer CRC32 checksums for each of the top level variables.

Summary: DAP4 is closer than DAP2 to the REST (Representational State Transfer) architecture, and it uses HATEOS (Hypermedia As The Engine Of Application State), making all of the server’s responses explicit via links in a document.

While DAP2 interwove the DAP and HTTP, using, for example, some of the HTTP headers as the only source of information that was critical to the DAP itself, DAP4 does not. Instead, DAP4 is completely isolated from HTTP, enabling it to work with other protocols without change. However, in as much as HTTP is a ubiquitous network transport protocol, the DAP4 specification includes a volume devoted solely to how a server should implement DAP4 web services using HTTP.

The REST interface for the protocol is described in Volume 2, Web Services, of the specification. DAP4 requires that a server implement at least three responses for each dataset: The DSR; DMR; and Data response. The DSR is a XML document that provides a capabilities response for the dataset. This document provides links to all of the other responses available for the dataset, along with other information. The DSR provides information about alternative encodings for the different responses in addition to enumerating the basic responses themselves. The DSR may also list server functions that may be used with/on the dataset.

DAP4 servers are encouraged to support HTTP content negotiation, providing the standard DSR, DMR and Data responses in a variety of forms.

Migrating from DAP2 to DAP4

The web service for DAP4 will likely need to be written from scratch, but the good news is that those are easy to write. For clients, the behavioral differences between DAP2 and DAP4 servers are small, with two exceptions. Since DAP4 optionally supports asynchronous responses, clients should be modified to access data available only using this new feature. DAP4 also supports content negotiation and that means a larger number of ways to get the different responses (even though each protocol has three basic responses).

Element and Attribute Names
Within the DMR XML document, it is assumed that XML element and XML attribute names are case sensitive.

Character Escapes Any string of characters appearing within an XML attribute in the DMR must apply the standard XML escapes. Specifically, any attribute value containing any of the following characters must replace them with the corresponding XML escape form.

So for example, given the occurrence of the attribute ‘name=“&<>”’ it must be re-written to this form ‘name=“&<>”’.

A name (aka identifier) in DAP4 consists of a sequence of any legal non-control UTF-8 characters. A control character is any UTF-8 character in the inclusive range 0x00 — 0x1F. Names are case sensitive.

Any name that begins with the character sequence “_” is considered reserved. Note that if the receiver encounters such a name and has no information on how to process the name, it may at its discretion either ignore the object with that name, or it may treat the name as an ordinary name.

A special case is when the “_” is followed by a reverse DNS name defining both the definer of that reserved name and possible additional naming information. This form of reserved name is preferred because it provides information about the organization that defined it.

A (reverse) DNS name is of this syntactic form.

An example might be edu.ucar.unidata.NAME1.NAME2…. This indicates the owner/definer of that name is edu.ucar.unidata and that the additional naming information (“NAME1.NAME2…) has meaning to the owner for defining the semantics of the so-named object.

Every object in a DAP4 Dataset has a Fully Qualified Name (FQN), which provides a way to unambiguously reference declarations in a dataset and which can be used in several contexts such as in the DMR in a constraint expression (see Section 1.8).

These FQNs follow the common conventions of names for lexically scoped identifiers. In DAP4 several kinds of lexical items provide lexical scoping: Dataset, Groups, Structures, Sequences, Enumerations, and AttributeSets. Just as with hierarchical file systems or variables in many programming languages, a simple grammar formally defines how the names are built using the names of the FQN’s components (see Section 1.10.2).

The FQN for a “top-level” variable — as opposed to e.g. a field in a structure or sequence — is defined purely by the sequence of enclosing groups plus the variable’s simple name. This also holds for Enumeration declarations.

Consider the following simple dataset, which contains a Structure named “inner” within a Structure named “outer” all contained in the Dataset “D”.

The FQN for the field ‘temperature’ is

Substituting the keyword Sequence for one or more occurrences of Structure in the above example will leave the FQNs unchanged. Note that the name of the dataset (“D”) is not included; it is implied by the leading “/”.

As is the case with Structure or Sequence variables, Groups can be nested to form hierarchies, too, and this example shows that case.

The FQN to the field temperature in the dataset shown is

Note the use of a different separator character . instead of / once we enter the scope of a structure (or sequence).

Enumeration constants are treated similarly to fields. Consider this example.

The FQN for the v1 constant in e is as follows.

Notes:

The characters / and . have special meaning in the context of a fully qualified name. This means that if a name is added to the FQN and that name contains either of those two characters, then those characters must be specially escaped so that they will not be misinterpreted. The defined escapes are as follows.

Note that the escape character itself must be escaped. Also note that this form of escape using ‘\’ is independent of any required XML escape (Section 1.5.1).

DAP4 imposes the rule that the definition of any object (e.g. dimension, group, or enumeration) must occur before any reference to that object. This rule also applies within a group, which in turn implies that, for example, all dimensions must be declared before all variables that reference them.

The declarations in a DMR can be grouped into two classes. One class is definitional. That is, it defines metadata that is used in the rest of the DMR. These definitional declarations are Groups (including the outer Dataset), Dimensions, and Enumerations. Such declarations do not contain data values themselves, although they may define constants such as the dimension size. The data-bearing declarations are Variables and Attributes. These elements of the data model are used to house data values or semantic metadata read from the dataset (or, in the latter case) synthesized from the values and standards/conventions that the dataset is known to follow.

Every DMR contains exactly one Dataset declaration. It is the outermost XML element of the DMR.

A dataset is specified using this XML form:

The name, dapVersion, and dmrVersion attributes are required. The attributes have the following semantics:

The body of the Dataset is the same as the body of a [1.5.8], and semantically the Dataset acts like the outermost, root, group.

A group is specified using this XML form:

A group defines a name space and contains other DAP elements. Specifically, it can contain in this order: dimension, enumerations, variables, and (sub-)groups. The fact that groups can be nested means that the set of groups in a DMR form a tree data structure. For any given DMR, there exists a root group that is the root of this tree.

A nested set of groups defines a variety of name spaces and access to the contents of a group is specified using a notation of the form /g1/g2/…/gn. This is called a path. By convention / refers to the root group (the Dataset declaration). Thus the path /g1/g2/g3 indicates that one should start in the root group, move to group g1 within that root group, then to group g2 within group g1, and finally to group g3. This is more fully described in the section on Fully Qualified names (Section [1.5.4]).

The order of declarations within a Group is fixed and must conform to this order.

For comparison purposes, DAP groups correspond to netCDF-4 groups and not to the more complex HDF5 Group type: i.e. the set of groups must form a tree.

Semantic Notes

A dimension declaration is specified using this XML form.

The size is a positive integer (which means that a zero length dimension is illegal). As described in the Arrays Section, the maximum size of any dimension is \$2^(61) - 1\$. A dimension declaration will be referenced elsewhere in the DMR by specifying its name. It should also be noted that anonymous dimensions also exist. They have a size but no name. Anonymous dimensions SHOULD NOT be declared.

Semantic Notes

An enumeration type defines a set of names with specific values called enumeration constants. As will be seen in Section [1.5.13], enumeration types may be used as the type for variables or attributes. The values that can be assigned to such typed objects must come from the set of enumeration constants.

An enumeration type specifies a set of named, integer constants. When a data source has a variable of type ‘Enumeration’ a DAP 4 server MUST represent that variable using a specified integer type, up to and including a 64-bit unsigned integer.

An Enumeration type is declared using this XML form.

Semantic Notes

The DAP4 specification assumes the existence of certain pre-defined, declared types called atomic types. As their name suggests, atomic data types are conceptually indivisible. Atomic variables are used to store integers, real numbers, strings and URLs. There are five classes of atomic types, with each family containing one or more variations: integer, floating-point, string, enumerations, and opaque.

Integer Types

The integer types are summarized in the following table. The syntax for integer constants is defined in Appendix 1.10.2.

Note that for historical reasons, the Char type is defined to be a synonym of UInt8, this mean that technically, the Char type has no associated character set encoding. However, servers and clients are free to infer typical character semantics to this type. The inferred character set encoding is chosen purely at the discretion of the server or client using whatever conventions they agree to use, possibly specified using attributes. Note specifically that multi-byte character encodings such as UTF-8 are problematic precisely because they can be multi-byte.

Floating Point Types

The floating-point data types are summarized in Table 2. The two floating-point data types use IEEE 754 [7] to represent values. The two types correspond to ANSI C’s float and double data types. The syntax for floating point constants is defined in Appendix 1.10.2.

String Types

The string data types are summarized in Table 3. Again, the syntax for these is defined in Appendix 1.10.2

Strings are individually sized. This means that in an array of strings, for example, each instance of that string MAY be of a different size.

The Opaque Type

The XML scheme for declaring an Opaque type is as follows.

The Opaque type is use to hold objects like JPEG images and other Binary Large Object (BLOB) data that have significant internal structure which might be understood by clients (e.g., an image display program) but that would be very cumbersome to describe using the DAP4 built-in types. Defining a variable of type Opaque does not communicate any information about its content, although an attribute could be used to do that.

Opaque instances are individually sized. This means that in an array of opaques, for example, each instance of that opaque MAY be of a different size.

Semantic Notes

The Enum Type

The XML scheme for declaring an Enum type is as follows.

The Enum type is intended to be used in the definition of a variable. It should not be confused with the definition of an Enumeration, but rather references such a definition.

Semantic Notes

A Note Regarding Implementation of the Atomic Types

When implementing the DAP, it is important to match information in a data source or read from a DAP response to the local data type which best fits those data. In some cases an exact match may not be possible. For example Java lacks unsigned integer types [6]. Implementations faced with such limitations MUST ensure that clients will be able to retrieve the full range of values from the data source. If this is impractical, then the server or client may implement this rule by hiding the variable in question or returning an error.

There are currently two container types: <Structure> and <Sequence>.

The Structure Type

A Structure groups a list of variables so that the collection can be manipulated as a single item. The variables in a Structure may also be referred to as “fields” to conform to conventional use of that term, but there is otherwise no distinction between fields and variables. The Structure’s fields MAY be of any type, including Structure or Sequence. The order of items in the Structure is significant only in relation to the serialized representation of that Structure.

The Sequence Type

A Sequence is intended to represent a sequence of instances of objects. Suppose that we have a sequence of this form.

The corresponding Structure object is obtained by substituting the Sequence keyword with Structure. Our above example then has this associated Structure.

The semantics of a sequence are that it represents a sequence of instances of the corresponding Structure. The length of the Sequence MAY be different for every instance of a Sequence. Consider this array of Sequence.

This represents an array of six (3 times 2) sequence instances. However, the length MAY be different for each of those six instances.

Note that the <Sequence> construct was introduced to replace the concept of variable length dimensions. It turns out that trying to treat variable length dimensions as dimensions causes significant conceptual and implementation difficulties. It is hoped that isolating such variable length objects syntactically is a better representation.

Semantic Notes

Each variable in a data source MUST have a name, a type and one or more values. Using just this information and armed with an understanding of the definition of the DAP data types, a program can read any or all of the information from a data source.

The DAP variables come in several different types. There are several atomic types, the basic indivisible types representing integers, floating point numbers and the like, and a container type – the Structure or Sequence type – that supports aggregation of other variables into a single unit. A container type may contain both atomic typed variable as well as other container typed variables, thus allowing nested type definitions.

The DAP variables describe the data when it is being transferred from the server to the client. It does not necessarily describe the format of the data inside the server or client. The DAP defines, for each data type described in this document, a serialized representation, which is the information actually communicated between DAP servers and DAP clients. The serialized representation consists of two parts: the declaration of the type and the serialized encoding of its value(s). The data representation is presented in Section [1.6.1].

Arrays

An Array is a multi-dimensional indexed data structure. An Array’s member variable MUST be of some DAP data type. Array indexes MUST start at zero. Arrays MUST be stored in row-major order (as is the case with ANSI C), which means that the order of declaration of dimensions is significant. The size of each Array’s dimensions MUST be given. The total number of elements in an Array is fixed as that given by the product of the size(s) of its dimension(s). Note that a dimension size of zero is illegal.

For practical reasons having to do with current hardware limitations, the total number of bytes allocated to an array must fit in an unsigned 64-bit integer. The largest atomic types currently defined in this document are the floating point double and the (U)Int64 integer types. This means that the practical limit on the total number of elements is \$2^(64) / 8 = 2^(61)\$. Thus the dimension indices will run from 0 to a maximum of \$2^(61) - 1\$. Of course this limit on the maximum number of elements also applies to the maximum dimension size since the total number of elements is the product of all the dimensions sizes of the array.

There is a prescribed limit of 64 on the number of of dimensions for a variable (i.e. its arity). This is actually larger than will occur in practice. Assuming a dimension must be at least 1 bit in size, this effectively limits the number of dimensions to 61.

Semantic Notes

Attributes

Simple attributes are defined using the following XML scheme.

or

Attributes may also serve as containers for other attributes (and other containers). In this case, no type is specified, only a name.

In DAP4, Attributes (not to be confused with XML attributes) are tuples with four components:

This differs slightly from DAP2 Attributes because the namespace feature has been added, although clients can choose to ignore it. For more about namespaces, refer to Section [1.5.15]. The intent of including the namespace information is to simplify interactions with semantic web applications where certain schemas or standards have formal definitions of attributes.

Attributes are typically used to associate semantic metadata with the variables in a data source. Attributes are similar to variables in their range of types and values, except that they are somewhat limited when compared to those for variables: they cannot use Structure or Sequence types.

Attributes defined at the top-level within a group are also referred to as “group attributes”. Attributes defined at the root group (i.e. Dataset) are “global attributes,” which many file formats such as HDF4 or netCDF formally recognize.

While the DAP does not require any particular Attributes, some may be required by various metadata conventions. The semantic metadata for a data source comprises the Attributes associated with that data source and its variables. Thus, Attributes provide a mechanism by which semantic metadata may be represented without prescribing that a data source use a particular semantic metadata convention or standard.

Semantic Notes

Arbitrary XML content

DAP4 supports an explicit type to hold “arbitrary XML” markup that provides a way for the protocol to transport information encoded in XML. This is useful for “annotating” meta-data with information more complex than simple attributes. This can be used, for example, for passing semantic web information, or for passing out-of-band information: e.g about the conversion from some other meta-data system into DAP4.

The form on an other XML declaration is as follows.

There are no <value/> elements because the value of otherXML is the xml inside the <otherXML>…</otherXML>. The text content of the otherXML element must be valid XML and must be distinct from the XML markup used to encode elements of the DAP4 data model (i.e., in a practical sense, the content of an <OtherXML> attribute will be in a namespace other than DAP4). XML content may appear anywhere that an attribute may appear.

Attribute and OtherXML Specification and Placement

Attribute and OtherXML declarations MAY occur within the body of the following XML elements: Group, Dataset, Dimension, Variable, Structure, Sequence, and Attribute.

All elements of the DMR – Dataset, Groups, Dimensions, Variables, and Attributes – can contain an associated Namespace element. The namespace’s value is defined in the form of an XML style URI string defining the context for interpreting the element containing the namespace. Suppose, hypothetically, that we wanted to specify that an Attribute is to be interpreted as a CF convention [15]. One might specify this as follows.

Note that this is not to claim that this is how to specify a CF convention [15].; this is purely illustrative.

There are two response formats that a server MUST provide to the client.

DMR-Only Response If the client requests only the DMR, then it is returned as a standard XML encoded document. If constraints were specified, then the returned DMR may differ from the full DMR in that, for example, meta-data about only variables specified in the constraint will be returned. The DMR-Only response MUST be self-contained. This means that all declarations directly or transitively mentioned in the selected variables must be included in the returned DMR. Additionally, all attributes associated with the included declarations MUST be included as well.

Data Response The DAP4 data response uses a format very similar to that used for DAP2; the data payload is broken into two pieces. The first part holds metadata describing the names and types of the variables in the response while the second part holds the values of those variables.

The metadata information, sent as part 1 of the Data Response, is the DMR limited to just those variables included in the response. The response, however, MUST be self-contained (in the DMR-Only sense). DAP attributes for all included declarations MUST be included, but MAY be ignored by the receiving client.

Part 2 of the response consists of the binary data for each variable in the order they are listed in the DMR given as the response preface. DAP4 uses a receiver makes it right encoding, so the servers MAY simply write out binary data as they store it with the exceptions that floating-point data must be encoded according to IEEE 754[7] and Integer data must use twos-complement notation for signed types. Clients are responsible for performing byte-swapping operations needed to compute using the values retrieved.

The Data Response is encoded using chunking scheme (see Section [1.6.2]). that breaks it into N parts where each part is prefixed with a chunk type and chunk byte count header. Chunk types include data and error types, making it simple for servers to indicate to clients that an error occurred during the transmission of the Data Response and (relatively) simple for clients to detect that error.

As with DAP2, the response described here is a document that can be stored on disk or sent as the payload using a number of network transport protocols, HTTP being the primary transport in practice. However, any protocol that can transmit a document can be used to transmit these responses. As such, all critical information needed to decode the response is completely self-contained.

In the rest of this section we will describe the Data Response in the context of DAP4 using HTTP as its transport protocol.

Format of the DMR Part

The first part (part is not to be confused with chunk) of the Data Response always contains the DMR. The Data Response, when DAP is using HTTP as a transport protocol, is the payload for an HTTP response. It is separated from the last of the HTTP response’s MIME headers by a single blank line, which MIME defines as a carriage return (ASCII character with byte value of 13) followed by a line feed (ASCII character with byte value of 10). This combination can be abbreviated as CRLF.

Format Related DMR Attributes
The DMR MAY contain attributes that reflect information from the serialized data. Specifically, the following attributes are defined.

Format of the Data Part The second part of the Data response consists of the serialized variables as specified by the data DMR. The variable serializations are concatenated to form a single binary dataset. If requested, each variable’s serialization is followed by a CRC32 checksum.

Relationship to the Chunking format The data response format is technically independent of the chunking format (see 1.6.1.3).

The assumption is that the DMR will be in a chunk of its own, the first chunk, and the serialized binary data will be in one or more additional chunks. This produces a format like this

In the above and in the following, the form ‘{xxx}’ is intended to represent any instance of the xxx.

How the Chunked Encoding Affects the Data Response Format

In a sense, the chunked encoding does not affect the format of the Data Response at all. Conceptually, the entire binary Data Response is built and then passed through a ‘chunking encoder’ transforming it into one that is broken up into a series of chunks. That ‘chunked document’ is then sent as the payload of some transport protocol, e.g., HTTP. In practice, that would be a wasteful implementation because a server would need to hold the entire response in memory. A better implementation would, for HTTP, write the initial parts of the HTTP response (its response code and headers) and then use a pipeline of filters to perform the encoding operations. The intent of the chunking scheme is to make it possible for servers to build responses in small chunks, and once they know those parts have been built without error, send them to the client. Thus a server should choose the chunk size to be small enough to fit comfortably in memory but large enough to limit the amount of overhead spent by the software that encodes and decodes those chunks. When an error is detected, the normal flow of building chunks and sending the data along is broken and an error chunk should be sent (See Section [2.3.4]).

Given a DMR and the corresponding data, the serialized representation is formally described in this section.

In these examples, spaces and newlines have been added to make them easier to read. The real responses are more compact. Since this proposal is just about the form of the response - and it really focuses on the BLOB part - there is no mention of ‘chunking.’ For information on how this BLOB will/could be chunked. see Section [1.7]. NB: Some poetic license used in the following and the checksums for single integer values seems silly, but these are really simple examples.

Note that there is semantic limitation in the definition of ‘chunk’: the number of bytes in the CHUNKDATA must be equal to SIZE.

selection expression
The entire expression passed to the server that is used to choose specific parts of a dataset.

subset
The act of choosing parts of a dataset based on the type of one or more of its variables. We define several types of subsetting operations as follows:

index subsetting
Choosing parts of an array based on the indexes of that array’s dimensions. This operation always returns an array of the same rank as the original, although the size of the return array will likely be smaller. Index subsetting uses the bracket syntax described subsequently, and is zero-based (u[1] refers to 2th element of u).

field subsetting
Choosing specific variables (fields) from the dataset. A dataset in DAP4 is made up of a number of variables and those may be Structures or Sequences that contain fields. Field subsetting uses the brace syntax described later. One or more fields can be specified using a semicolon (;) as the separator.

filter
A filter is a predicate that can be used to choose sequence rows based on the values of fields of the sequence. the vertical bar (|) is used as a prefix operator for the filter predicate. Filters can be applied to fields of a Sequence. A filter predicate consists of one or more filter subexpressions. One or more subexpressions can be specified, using a comma (,) as the separator. Implicitly, multiple filter subexpressions are logically and’ded together.

filter subexpression
A simple expression that consists of a single variable/field; the expression is composed from traditional set of binary and unary operators: comparison operators (=, !=, <, ⇐, >, >=) for numbers and strings, and a string specific regular expression comparison operator (~=). The operands of the operators must be either numeric or string constants or a field of the Sequence. Specifically, only atomic-valued, scalar fields can be used in the filter expression.

id
The name of a variable. These must be absolute, with some specific exceptions. Absolute names are fully qualified names (See Section [1.5.4]).

The simplest constraint is the null string and it means ‘return everything’ from the dataset. Choosing variables in a dataset is referred to as the subset. To choose a subset of the variables in a dataset, enumerate them in a semicolon-separated list. To choose parts of a Structure, name those parts explicitly using the syntax

structure_name\{field name}

or

structure_name.field name

Each DAP4 dataset contains one or more Groups. The top-level Group is always present and is named / (pronounced root).

Subsetting fixed-size arrays in their index space is accomplished using square brackets. The syntax closely follows that of DAP2, with some extensions. For an array with N dimensions, N sets of brackets are used, even if the array is only subset on some of the dimensions. The names of array variables are fully qualified names (FQNs) so it’s possible to name arrays in structures and/or Groups. Array index values are zero-based as with a number of programming languages such as C and Java. Every array has a known starting index value of zero. Within the square brackets, several subexpressions are allowed:

[ ]
return all of elements elements for a particular dimension or apply a shared dimension slice (more on this later).

[ n ]
return only the value at a single index, where \$0 <= n < N\$ for a dimension of size N. This slicing operator does not reduce the dimensionality of an array, but does return a dimension size of one for the dimension to which this is applied.

[ start : step : last ]
return every value whose index is in the range start \$<=\$ index \$<=\$ last and where (index - start)% step == 0. NOTE: range defined by square brackets is inclusive to right (subset will include the last index).

[ start : last ]
return the values whose index is in the range start \$<=\$ index \$<=\$ last where step=1 by default.

[ start : ]
return every value whose index is in the range start \$<=\$ index \$<= N-1\$, where N is the dimension size. step=1 by default.

[ start : step : ]
return every value whose index is in the range start \$<=\$ index \$<= N-1\$, where N is the dimension size and where (index - start)% step == 0.

Subsetting can be applied to any array. It can also be applied to a scalar, but in this case, the only legal forms are [0] or [].

As a new feature in DAP4 constraints, index subset within square brackets can contain multiple, disjoint slices, where each slice is of any of the previously defined slice formats (most generally start:stride:last where last is included). The disjoint slices are separated by commas.

Using the preceding example dataset vol_1_ce_4, some disjoint index examples might be as follows.

The Sequence type is more general data type in DAP4 than in DAP2 where it was significantly limited. In DAP4 Arrays of Sequences will be supported as will Sequence fields that are themselves Arrays or Sequences. A Sequence variable is conceptually like a table of rows where each field in the Sequence is a column in the table (or like an array of Structures, where the size of the single array dimension is a secret). Note that while there is a big difference between the value held by a Structure and a Sequence, each has the same subsetting syntax in the CE. The differences is that Sequences may have filters applied during CEs while Structures may not.

Shared Dimensions provide additional information to indicate that a group of arrays share certain relationships; that specific groups of the arrays form coverages by indicating how dimensions of Maps and Arrays are linked. The DAP4 CE syntax provides a way to slice a Shared Dimension so that slice can be used by all of the arrays that use it without repeating the slicing operation for each Array. The syntax can be read ‘Assign the shared dimension X this slice,’ where the slice looks like, for example, row=[10:19]. All of the variations of the slice operator possible for an array are accepted for shared dimension slicing. In any CE, all of the shared dimension slicing clauses must precede the variable subsetting clauses.

Note DAP4 uses XML for it’s actual grammar, and because that’s wordy this document includes a mock notation. I will extend that notation used so far so it includes concepts needed to mimic DAP4’s notation for a coverage:

NOTE:

Using Shared Dimensions for array slicing adds some complexity to the processing of constraints. Two cases are important to consider and are shown in the examples.

The examples make these two cases clearer.

When a DAP4 server receives a request for a Data response, it must build and return a Data Response Document that contains a text/xml part containing a DMR, a separator and a binary part that contains the data values. The organization of the Data Response Document is described in detail elsewhere in this document. In this section the focus is on the DMR returned in the first part of the response and how it relates to the DMR for the original unconstrained dataset. We refer to the original dataset’s DMR as the DMR and the DMR associated with the data response as the CDMR (short-hand for Constrained DMR), although a data response can be generated using a null CE, we consider that a constraint, too.

The DMR contains a number of declarations for the dataset: Enumerations, Dimensions, Attributes, Groups and Variables. Each DMR and CDMR must follow the rules for the DMR described in this specification and, because DAP4 is a stateless protocol, each response from a server must stand on its own. Since a Constraint Expression alters the data returned (limiting variables, changing the size of dimensions and so on), it stands to reason that the contents of the CDMR will vary for any given dataset based on the CE. Furthermore, a goal of DAP4 is to specify that the CDMR be ‘minimal’ containing no unused definitions.

Because filters alter the values of variables, but not whether a variable is returned, they have no affect on the CDMR. Only the subsetting operators will be discussed here.

While subsetting provides ways to choose data based on the dataset structure and the types of the variables, filters provide a way to choose data based on their values. The values to be returned are denoted using one or more simple predicates. The general syntax for a filter expression is to follow a subset (projection) expression with a pipe (|) and one or more filter predicates. Multiple predicates are separated by commas and the value of complete predicate is the logical AND of the comma-separated subexpressions.

Filter expressions can only be applied to Sequence variables (or arrays of them). In each case the result of the filter operation returns the same type variable. A Sequence variable is essentially a table of values and thus can be thought of as containing a number of rows and the filter expression is applied to each row in the order those rows are provided to the expression evaluator. Every row that satisfies the predicate will be included in the value returned; those that don’t will not be included in the result. Note that no new values are computed by these operations; no interpolations, means, etc., are performed.

The behavior of filtering expressions on Sequences will be covered in the following sections.

The basic syntax for filters is that there is a subsetting expression, a pipe (|) and then one or more filter predicates. This syntax can appear any place a selection expression can appear, so it can be used inside braces when an Array or Sequence is a field of a Structure or Sequence. Note that the filter expression prefix operator binds to the index subset immediately to its left at the same level (i.e. eliding braces). Some examples follow.

This RELAX NG grammar is the definitive formal grammar for the DMR.

Within the RELAXNG DAP4 grammar there are markers for occurrences of primitive type such as integers, floats, or strings (ignoring case). The markers typically look like this when defining an attribute that can occur in the DAP4 DMR.

The “<data type=`"dap4_string”/>” specifies the lexical class for the values that this attribute can have. In this case, the "`Principal_Investigator” attribute is defined to have a DAP4 string value. Similar notation is used for values occurring as text within an xml element.

The lexical specification later in this section defines the legal lexical structure for such items. Specifically, it defines the format of the following lexical items.

The specification is written using the extended POSIX regular expression notation [11] with some additions.

Notes:

The RELAXNG [13] grammar references the following specific types. For each type, the following table give the lexical format as defined by the patterns previously given or by specific patterns as listed.

Note that the above lexical element classes are not disjoint. The type element “<datatype=…/>” should be sufficient to interpret the type within the DMR.

The UTF-8 specification [14] defines several ways to validate a UTF-8 string of characters.

The full (most correct) validating version of UTF8 character set is as follows.

The lines of the above expression cover the UTF-8 characters as follows: 1. non-overlong 2-byte 2. excluding overlongs 3. straight 3-byte 4. excluding surrogates 5. straight 3-byte 6. planes 1-3 7. planes 4-15 8. plane 16

Note that values from 0 through 127 (ASCII and control characters) are not included in any of these definitions.

The above reference also defines some alternative regular expressions. First, there is what is termed the partially relaxed version of UTF8 defined by this regular expression.

Second, there is what is termed the most-relaxed version of UTF8 defined by this regular expression.

Any conforming DAP4 implementation MUST use at least the most-relaxed expression for validating UTF-8 character strings, but MAY use either the partially-relaxed or the full validation expression.

It is conventient to have a Bison grammar that corresponds to the above RELAX NG grammar. If there is a conflict, then the RELAX NG grammar is considered correct.

The standard DAP4 dataset resource roles and encodings (plus a few alternate encodings) that are defined in the core DAP4 documents are:

Dataset Services Response (DSR)