Description of the preferred data formats (XML and netcdf)

We prefer to use netcdf to store all array-based data and XML to store tabular data. Since netcdf has no built-in compression, it is sometimes advantageous to store only non-missing data. For this purpose, we use the SparseGrid formats.  Regardless of how the data are stored, after the data have been written, it is necessary to notify about the presence of the data using the Index::Record.

We rely on a few global attributes -- the origin location and time, for example. The Time is the UNIX time (seconds since Jan. 1, 1970) with any fractional part. Attributes are optional pieces of information that may be attached to the data. They are not required, but if you specify them, you need to use the names we understand and specify the units that they are provided in.
When creating NetCDF files, make sure to specify the dimensions of the data and to specify all the variables.

Index::Record format for netcdf files

You should write out an Index Record for each netcdf file. The format of the Index Record is of the form: 
  <item>
    <time fractional="0.000000"> 799875952 </time>
    <params>netcdf complete-file-name </params>
    <selections>1995:05:07-19:45:52 Velocity 0.468750 </selections>
  </item>
It is recommended that you compress the file using gzip after writing it out. If compressed using gzip, the file should have a .gz extension. It is also recommended that the file name written to the record have the machine name prepended, i.e. of the form 
     vortex:/tmp/netcdf/Reflectivity_0.47_19950507-194552.netcdf.gz
A relative path to the index can be provided by specifying, for example: 
     {indexlocation} Reflectivity/00.50/19950507-194552.netcdf.gz
The selections together form an unique way to refer to the product just written out. For example, the date, the product name and elevation angle in the example shown. The selections should be of the form: 
   time-string  TypeName  sub-type
The sub-type is optional. The DataType should match the global TypeName attribute in the file. The time-string may be in any format.

Index::Record format for xml files

You should write out an Index Record for each XML file. The format of the Index Record is of the form: 
  <item>
   <time fractional="0.000000"> 799875952 </time>
   <params>W2ALGS GzippedFile vortex:/tmp/xmldir xmldata xmlfilename.gz </params>
   <selections>1995:05:07-19:45:52 MDA 0.5 </selections>
   </item>
It is recommended that you compress the file using gzip after writing it out. If compressed using gzip, the second parameter should be a "GzippedFile" (other options include "FlatFile" and "BzippedFile"). It is also recommended that the directory name written to the record have the machine name prepended, i.e. of the form 
     vortex:/tmp/xmldir/
You may also provide a relative path using the 
 {indexlocation}
identifier.

The selections together form an unique way to refer to the product just written out. For example, the date, the product name and elevation angle in the example shown. The selections should be of the form: 

   time-string  TypeName  sub-type
The sub-type is optional. The DataType should match the global TypeName attribute in the file. The time-string may be in any format.

Cartesian Grid Netcdf format

The Cartesian Grid is tangential to the earth's surface at the center. The origin Latitude, Longitude and Height are those of the center of the grid, while the Xcorner is that of the north-west corner and the Ycorner is the south-east corner. By specifying these three locations and the grid dimension, you specify the grid spacing (assumes a uniform grid). 
dimensions:
        Xdirection = 501 ;
        Ydirection = 501 ;
variables:
        float VIL(Xdirection, Ydirection) ;
                VIL:Units = "SI" ;
 
// global attributes:
                :TypeName = "VIL" ;
                :DataType = "CartesianGrid2D" ;
                :Latitude = 32.5730555555555 ;
                :Longitude = -97.3030555555556 ;
                :Height = 228.000000000065 ;
                :Time = 799875952 ;
                :FractionalTime = 0. ;
                :attributes = " ExpiryInterval vcp" ;
                :ExpiryInterval-unit = "Minutes" ;
                :ExpiryInterval-value = "15" ;
                :vcp-unit = "dimensionless" ;
                :vcp-value = "21" ;
                :XCornerLat = 30.2989100884002 ;
                :XCornerLon = -94.7007372622298 ;
                :XCornerHt = 10055.5421673762 ;
                :YCornerLat = 34.7908143153896 ;
                :YCornerLon = -100.039082234929 ;
                :YCornerHt = 10055.5421673771 ;
                :OriginLat = 30.2989100884002 ;
                :OriginLon = -99.9053738488813 ;
                :OriginHt = 10055.5421673771 ;
                :MissingData = -99900. ;
                :RangeFolded = -99901. ;
data:

Radial data Netcdf format

A single elevation (sweep) of data is stored as a collection of radials. These radials are assumed to be non-uniformly spaced in azimuthal space, so the starting azimuth of the radial beam is written out, as well as the beam width of each beam. The gate width is assumed constant for each beam. The location is the location of the radar. 
dimensions:
        Azimuth = 367 ;
        Gate = 460 ;
variables:
        float Azimuth(Azimuth) ;
                Azimuth:Units = "Degrees" ;
        float BeamWidth(Azimuth) ;
                BeamWidth:Units = "Degrees" ;
        float GateWidth(Azimuth) ;
                GateWidth:Units = "Meters" ;
        float Reflectivity(Azimuth, Gate) ;
                Reflectivity:Units = "dBZ" ;
 
// global attributes:
                :TypeName = "Reflectivity" ;
                :DataType = "RadialSet" ;
                :Latitude = 32.573055267334 ;
                :Longitude = -97.3030548095703 ;
                :Height = 227.999999999916 ;
                :Time = 799875952 ;
                :FractionalTime = 0. ;
                :attributes = " ExpiryInterval NyquistVelocity vcp radarName" ;
                :ExpiryInterval-unit = "Minutes" ;
                :ExpiryInterval-value = "15" ;
                :NyquistVelocity-unit = "MetersPerSecond" ;
                :NyquistVelocity-value = "53" ;
                :vcp-unit = "dimensionless" ;
                :vcp-value = "21" ;
                :radarName-unit = "dimensionless" ;
                :radarName-value = "KTLX" ;
                :Elevation = 0.46875 ;
                :ElevationUnits = "Degrees" ;
                :MissingData = -99900. ;
                :RangeFolded = -99901. ;
The attributes are all optional. NyquistVelocity may be defined as either an attribute (valid for the whole elevation), or as a radial-by-radial variable: 
      float NyquistVelocity(Azimuth) ;
               NyquistVelocity:Units = "MetersPerSecond";

LatLonGrid netcdf format

The origin of the Lat-Lon-Grid is at the northwest corner (this is 0,0).

dimensions:
        Lat = 650 ;
        Lon = 700 ;
variables:
        float SHI(Lat, Lon) ;
                SHI:Units = "dimensionless" ;

// global attributes:
                :TypeName = "SHI" ;
                :DataType = "LatLonGrid" ;
                :Latitude = 37. ;
                :Longitude = -100. ;
                :Height = 1000. ;
                :Time = 990402843 ;
                :FractionalTime = 0.475000000005821 ;
                :attributes = "" ;
                :LatGridSpacing = 0.01 ;
                :LonGridSpacing = 0.01 ;
                :MissingData = -99900.f ;
                :RangeFolded = -99901.f ;

SparseGrid netcdf format

If your data are sparse, you can save a whole bunch of space by saving only the non-missing data in a simple run-length encoded format. The global attributes and the other variables are the same as the regular formats. However,  the dimensions are provided by pixel_x, pixel_y for the first two dimensions and run_length for the number of values in a run (default=1 if run_length is omitted).  The original dimensions are also required even if no variables actually use them. The default background value is MISSING_DATA unless a different value is provided (see the SparseRadialSet example below).

Example of a sparse lat-lon-grid:
dimensions:
        Lat = 650 ;
        Lon = 700 ;
        pixel = 23541 ;
variables:
        float Reflectivity_0C(pixel) ;
                Reflectivity_0C:Units = "dBZ" ;
        short pixel_x(pixel) ;
        short pixel_y(pixel) ;
        int pixel_count(pixel) ;

// global attributes:
                :TypeName = "Reflectivity_0C" ;
                :DataType = "SparseLatLonGrid" ;
                :Latitude = 37. ;
                :Longitude = -100. ;
                :Height = 0. ;
                :Time = 990376569 ;
                :FractionalTime = 0.584999999999127 ;
                :attributes = " ColorMap Unit" ;
                :ColorMap-unit = "dimensionless" ;
                :ColorMap-value = "Reflectivity" ;
                :Unit-unit = "dimensionless" ;
                :Unit-value = "dBZ" ;
                :LatGridSpacing = 0.01 ;
                :LonGridSpacing = 0.01 ;
                :MissingData = -99900.f ;
                :RangeFolded = -99901.f ;

Example of a sparse RadialSet:

dimensions:
        Azimuth = 360 ;
        Gate = 460 ;
        pixel = 4673 ;
variables:
        float Azimuth(Azimuth) ;
                Azimuth:Units = "Degrees" ;
        float BeamWidth(Azimuth) ;
                BeamWidth:Units = "Degrees" ;
        float GateWidth(Azimuth) ;
                GateWidth:Units = "Meters" ;
        float PrecipConfidence(pixel) ;
                PrecipConfidence:Units = "dimensionless" ;
        short pixel_x(pixel) ;
        short pixel_y(pixel) ;
        int pixel_count(pixel) ;

// global attributes:
                :TypeName = "PrecipConfidence" ;
                :DataType = "SparseRadialSet" ;
                :Latitude = 35.3330001831055 ;
                :Longitude = -97.2779998779297 ;
                :Height = 384. ;
                :Time = 1122583396 ;
                :FractionalTime = 0.947000000000116 ;
                :attributes = " BackgroundValue ColorMap Unit" ;
                :BackgroundValue-unit = "dimensionless" ;
                :BackgroundValue-value = "0" ;
                :ColorMap-unit = "dimensionless" ;
                :ColorMap-value = "Fuzzy" ;
                :Unit-unit = "dimensionless" ;
                :Unit-value = "dimensionless" ;
                :Elevation = 0. ;
                :ElevationUnits = "Degrees" ;
                :RangeToFirstGate = 0. ;
                :RangeToFirstGateUnits = "Meters" ;
                :MissingData = -99900.f ;
                :RangeFolded = -99901.f ;

Example of a WindField netcdf format

This is essentially a 2 LatLonGrid in one file.  Providing the meanwind is recommended, but not required.

dimensions:
        lat = 725 ;
        lon = 775 ;
variables:
        float uArray(lat, lon) ;
                uArray:Units = "MetersPerSecond" ;
        float vArray(lat, lon) ;
                vArray:Units = "MetersPerSecond" ;

// global attributes:
                :TypeName = "KMeansMotionEstimate" ;
                :DataType = "WindField" ;
                :Latitude = 37.5 ;
                :Longitude = -101. ;
                :Height = 0. ;
                :Time = 1129321566 ;
                :FractionalTime = 0.323000030519324 ;
                :attributes = " Unit meanwind" ;
                :Unit-unit = "dimensionless" ;
                :Unit-value = "MetersPerSecond" ;
                :meanwind-unit = "dimensionless" ;
                :meanwind-value = "-0.00318577" ;
                :LatGridSpacing = 0.00999999977648258 ;
                :LonGridSpacing = 0.00999999977648258 ;
                :MissingData = -99900.f ;
                :RangeFolded = -99901.f ;

Tabular data

Tabular data are written out in XML. As with netcdf data, there are a number of attributes that require, to describe each Table as well as to describe each row.

Each section is described below.

The entire table is enclosed in a datatable tag and has two child elements: datatype and data: 

  <datatable>
    <datatype/>
    <data/>
  </datatable>

The datatype tag

The datatype tag describes the table as a whole. It has one stref tag, which describes the location and time of the table as a whole. It also has any number of attr tags, which refer to attributes of the table. 
 <datatype name="MDA" >
  <stref>
   <location>
    <lat>
     <angle units="Degrees" value="32.5731" />
    </lat>
    <lon>
     <angle units="Degrees" value="-97.3031" />
    </lon>
    <ht>
     <length value="228" units="Meters" />
    </ht>
   </location>
   <time units="secondsSinceEpoch" value="799877001" />
  </stref>
  <attr name="ExpiryInterval" >
   <datacolumn units="Minutes" name="ExpiryInterval" >
    <item value="15" />
   </datacolumn>
  </attr>
 </datatype>
You can associate a column of values with a single attribute, although most commonly, you would associate only one value.

The data tag

Tabular data consists of a number of columns. Each column would be described with subtag describing the value for each row: 
 <data>
  <datacolumn units="dimensionless" name="AlgRank" >
   <item value="0" />
   <item value="1" />
   <item value="2" />
   <item value="3" />
  </datacolumn>
  <datacolumn units="Meters" name="Base" >
   <item value="1438" />
   <item value="2760" />
   <item value="3760" />
   <item value="2892" />
  </datacolumn>
 </data>
The column named "AlgRank" is special -- it describes the relative importance of the rows, with 0 being the most important. It may be omitted, if all the rows are equally important.

Contour Data in XML

The general format is as follows:
<contours>
  <datatype> ... </datatype>
  <contourdata>
    <array length="...">
        <contour> ... </contour>
        <contour> ... </contour>
        ...
    </array>
  </contourdata>
</contours>

The data type documentation is described above, in the section on Tabular data.
The contour is an array of locations; so a contour tag is organized as follows:
<contour>
  <datatype> ... </datatype>
  <locationdata>
     <array length="...">
         <location> ... </location>
     </array>
  </locationdata>
</contour>
In the datatype element of the contour tag, you could provide an attribute named "Color"
so that different contours get different colors.