Building and Testing DMR++ Documents

OPeNDAP’s DMR++ software does not currently support netCDF3 files.^[3] A complicating factor in building DMR++ documents is that it can be hard to tell at a glance if a file is netCDF version 3 or version 4. A file with the suffix .nc4 is conventionally recognized as a netCDF-4 file. However, the file suffix .nc is ambiguous since it is often used for both netCDF-3 and netCDF-4 files.

You can use the ncdump command to determine if a netCDF file is either classic netCDF-3 or netCDF-4 (You can learn more in the NetCDF documentation here). Here are two files, both using the suffix .nc where the first is netCDF3 and the second is netCDF4.

You have:

In this section we jump right into some examples without much explanation. This shows the minimum amount of work needed to build the DMR++ and sidecar files. See Section 3.2.5 for details about the gen_dmrpp_side_car command, which is the recommended command for building DMR++ documents (April 2025).

Change to the directory that holds your data files and assign an environment variable to the full pathname of that directory. This will streamline some of the later steps in this section. In my case that directory is called HDF4-dir, and I used the environment variable 'DATA.'

Here are the files on my computer in the directory assigned to $DATA

Run the Docker container. The docker run command returns the Container ID (a long hexadecimal string) when the -d (run a detached container) is used. The --name option sets hyrax as the name of the container which will be used in later commands. Running the container this way enables us to use both build DMR++ documents and later test them.

If you want to use the latest version of the gen+dmrpp_side_car command, replace the version number in opendap/hyrax:1.17.1-126 with snapshot. Using opendap/hyrax:snapshot will always get the most recent version of the software.

To build a DMR++ for the first AIRS file we can run the gen_dmrpp_side_car command, using docker exec, with the file’s name. Because this file is an HDF4 file, the command option -H is used.

In this second example both the DMR++ and a sidecar missing data file (3B42.19980101.00.7.HDF_mvs.h5) are built. As is often the case, the DMR++ and missing data files together are only 2% of the data file’s size.

This is also an HDF4 file, so the -H option is used.

The final example in this section shows building a DMR++ for an HDF5 file. For an HDF5 file, do not include the -H option.

A DMR++ document is an eXtensible Markup Language (XML) document. We call the data file/granule that the DMR++ describes the source data file. Each DMR++ has at least one source data file, but may have more than one source data file, for example, with HDF4-EOS2 geo-referencing data. The first XML element in the DMR++ contains a URL that points to the DMR++ document’s source data file. It looks like this:

There are three XML attributes in the root element of the DMR++ that are relevant to this discussion. They are:

The value of the dmrpp:href attribute is the source of data values that the Hyrax data server will use with building data responses. This URL can be either an HTTP, HTTPS or file:// URL (for more about the latter option, see Section 4.

However, when the OPeNDAP DMR++ was first developed for use by NASA ESDIS, we did not want to encode the URl to the data file into the DMR++. Instead, we planned on using the ESDIS Common Metadata Repository (CMR) to look up information about a granule and use that to find the source data file. This helped guard against having to edit many of the documents while the ESDIS system was in flux (i.e., it was a decision well aligned with agile development principles). In place of an explicit URL to the source data file, the gen_dmrpp_side_car will, by default, use a template string that the hyrax data server substitutes at runtime with the current data source URL as read from CMR.

What if you do not need or want that? The -u option of gen_dmrpp_side_car provides a way to tell the DMR++ document builder to use a specific value for the data source URL. The following examples show the DMR++ XML with the template value for the data source URL and then using a URL set with the -u option.

The template value for the data source URL is OPeNDAP_DMRpp_DATA_ACCESS_URL

The -u option provides the literal text for the value of the dmrpp:href XML attribute.

The mechanism described above for the data source URL, where the DMR++ builder provides a template value unless overridden using the -u option, applies similarly to references for sidecar geo-referencing data. By default, the name of the sidecar file itself is used. To include a template value (OPeNDAP_DMRpp_SC_DATA_ACCESS_URL) instead, the -U option must be specified. As with the data source URL, the -s option (described below in Section 3.2.5) may be used to explicitly set the sidecar file URL.

There is one exception to the rule that -u is used for the data source URL and -s is used for the sidecar data file. If -u is used, that name will be used as a pattern for the sidecar data file such that the missing data file will be assumed to be named the same as the data source, but with the suffix _mvs.h5.

In this example, we show the three files made from an HDF4-EOS2 file that where the sidecar file is necessary. The output of the command is shown first, followed by two views inside the DMR++ document.

The gen_dmrpp_side_car command takes a few options that control how it builds DMR++ and sidecar files.

In the Section 3.2.2 we used one docker command to start a container and then a second docker command to run the DMR++ builder inside that container. Here is an explanation of those commands in more detail. First, the container is started on the host computer.

The docker run -d …​ command will run the Hyrax container on your computer (called the host computer) in detached mode. The Hyrax container includes both the complete Hyrax service and the gen_dmrpp_side_car command. Later this server will be used to test the DMR++ documents that are built.

The volume mount, from $DATA to /usr/share/hyrax mounts the current directory of the host computer running the container to the directory /usr/share/hyrax inside the container. That directory is the root of the Hyrax server’s data tree. This means that the data files in the $DATA directory will be accessible by the server running in the container without any other configuration.

Complete option summary:

Once running, the container is used to run the command that will build the DMR++ document.

The command that built the DMR++ (and sidecar) file really consists of two commands. The first is docker exec -it -w /usr/share/hyrax hyrax which instructs docker to execute a program in the running container named hyrax and do so by first changing to the directory /usr/share/hyrax in that container. By using the -w option we are able to run the gen_dmrpp_side_car command in the directory within the container where data appear.

The second command instructs the docker container to run gen_dmrpp_side_car using the arguments -i 3B42.19980101.00.7.HDF -H -U which mean use the file 3B42.19980101.00.7.HDF as the input data file, assume it is an HDF4 file and use the template name for the sidecar data file.

Complete option summary for the docker exec command:

Make sure you are in the directory with the HDF4 files for these steps.

Get the command to return its help information:

will return:

Let’s build a DMR++ now, by explicitly using the container:

starts the bash shell in the container, with the current directory as root (/)

Change to the directory that is the root of the data (you’ll see your HDF4 files in here):

You will see, roughly:

In that directory, use the get_dmrpp_h4 command to build a DMR++ document for one of the files:

Copy that pattern for whatever file you use. From the /usr/share/hyrax directory, you pass get_dmrpp_h4 the name of the file (because it’s local to the current directory) using the -i option. The -u option tells the command to embed the URL that follows it in the DMR++. I’ve used a file:// URL to the file /usr/share/hyrax/3B42.19980101.00.7.HDF.

Building the DMR++ and embedding a file:// URL will enable testing the DMR++.

Here is a simple shell command that you can run on the host computer that will eliminate most of the above.

Start the container:

Check if it is running:

The command, written for the Bourne Shell, is:

Copy that, save it in a file (I named the file get_dmrpp_h4.sh).

Run the command on the host, not the docker container, and in the directory with the HDF4 files (you don’t have to do that, but sorting out the details is left as an exercise for the reader. Run the command like this:

The DMR++ will appear when the command completes.

The get_dmrpp application supports both S3 hosted granules as inputs, and uploading generated DMR++ files to an S3 bucket.

If your group is already serving data with Hyrax and the data representations that are generated by your Hyrax server are satisfactory, then a careful inspection of the localized configuration, typically held in /etc/bes/site.conf, will help you determine what configuration state you may need to inject into get_dmrpp.