Description of the preferred data formats (XML and netcdf)
We prefer to use netcdf to store all array-based data and XML to store tabular data. Since netcdf has no built-in compression, it is sometimes advantageous to store only non-missing data. For this purpose, we use the SparseGrid formats. Regardless of how the data are stored, after the data have been written, it is necessary to notify about the presence of the data using the Index::Record.We rely on a few global attributes -- the origin location and time, for example. The Time is the UNIX time (seconds since Jan. 1, 1970) with any fractional part. Attributes are optional pieces of information that may be attached to the data. They are not required, but if you specify them, you need to use the names we understand and specify the units that they are provided in.
When creating NetCDF files, make sure to specify the dimensions of the data and to specify all the variables.
Index::Record format for netcdf files
You should write out an Index Record for each netcdf file. The format of the Index Record is of the form:<item>It is recommended that you compress the file using gzip after writing it out. If compressed using gzip, the file should have a .gz extension. It is also recommended that the file name written to the record have the machine name prepended, i.e. of the form
<time fractional="0.000000"> 799875952 </time>
<params>netcdf complete-file-name </params>
<selections>1995:05:07-19:45:52 Velocity 0.468750 </selections>
</item>
vortex:/tmp/netcdf/Reflectivity_0.47_19950507-194552.netcdf.gzA relative path to the index can be provided by specifying, for example:
{indexlocation} Reflectivity/00.50/19950507-194552.netcdf.gz
The selections together form an unique way to refer to the
product just written out. For example, the date, the product
name and elevation angle in the example shown. The selections
should be of the form:
time-string TypeName sub-typeThe sub-type is optional. The DataType should match the global TypeName attribute in the file. The time-string may be in any format.
Index::Record format for xml files
You should write out an Index Record for each XML file. The format of the Index Record is of the form:<item>It is recommended that you compress the file using gzip after writing it out. If compressed using gzip, the second parameter should be a "GzippedFile" (other options include "FlatFile" and "BzippedFile"). It is also recommended that the directory name written to the record have the machine name prepended, i.e. of the form
<time fractional="0.000000"> 799875952 </time>
<params>W2ALGS GzippedFile vortex:/tmp/xmldir xmldata xmlfilename.gz </params>
<selections>1995:05:07-19:45:52 MDA 0.5 </selections>
</item>
vortex:/tmp/xmldir/You may also provide a relative path using the
{indexlocation}
identifier.
The selections together form an unique way to refer to the product just written out. For example, the date, the product name and elevation angle in the example shown. The selections should be of the form:
time-string TypeName sub-typeThe sub-type is optional. The DataType should match the global TypeName attribute in the file. The time-string may be in any format.
Cartesian Grid Netcdf format
The Cartesian Grid is tangential to the earth's surface at the center.
The origin Latitude, Longitude and Height are those of the
center of the grid, while the Xcorner is that of the north-west corner
and the Ycorner is the south-east corner.
By specifying these three locations and the grid dimension, you specify
the grid spacing (assumes a uniform grid).
dimensions:
Xdirection = 501 ;
Ydirection = 501 ;
variables:
float VIL(Xdirection, Ydirection) ;
VIL:Units = "SI" ;
// global attributes:
:TypeName = "VIL" ;
:DataType = "CartesianGrid2D" ;
:Latitude = 32.5730555555555 ;
:Longitude = -97.3030555555556 ;
:Height = 228.000000000065 ;
:Time = 799875952 ;
:FractionalTime = 0. ;
:attributes = " ExpiryInterval vcp" ;
:ExpiryInterval-unit = "Minutes" ;
:ExpiryInterval-value = "15" ;
:vcp-unit = "dimensionless" ;
:vcp-value = "21" ;
:XCornerLat = 30.2989100884002 ;
:XCornerLon = -94.7007372622298 ;
:XCornerHt = 10055.5421673762 ;
:YCornerLat = 34.7908143153896 ;
:YCornerLon = -100.039082234929 ;
:YCornerHt = 10055.5421673771 ;
:OriginLat = 30.2989100884002 ;
:OriginLon = -99.9053738488813 ;
:OriginHt = 10055.5421673771 ;
:MissingData = -99900. ;
:RangeFolded = -99901. ;
data:
Radial data Netcdf format
A single elevation (sweep) of data is stored as a collection of
radials.
These radials are assumed to be non-uniformly spaced in azimuthal
space,
so the starting azimuth of the radial beam is written out, as well as
the beam width of each beam. The gate width is assumed constant for
each beam. The location is the location of the radar.
dimensions:The attributes are all optional. NyquistVelocity may be defined as either an attribute (valid for the whole elevation), or as a radial-by-radial variable:
Azimuth = 367 ;
Gate = 460 ;
variables:
float Azimuth(Azimuth) ;
Azimuth:Units = "Degrees" ;
float BeamWidth(Azimuth) ;
BeamWidth:Units = "Degrees" ;
float GateWidth(Azimuth) ;
GateWidth:Units = "Meters" ;
float Reflectivity(Azimuth, Gate) ;
Reflectivity:Units = "dBZ" ;
// global attributes:
:TypeName = "Reflectivity" ;
:DataType = "RadialSet" ;
:Latitude = 32.573055267334 ;
:Longitude = -97.3030548095703 ;
:Height = 227.999999999916 ;
:Time = 799875952 ;
:FractionalTime = 0. ;
:attributes = " ExpiryInterval NyquistVelocity vcp radarName" ;
:ExpiryInterval-unit = "Minutes" ;
:ExpiryInterval-value = "15" ;
:NyquistVelocity-unit = "MetersPerSecond" ;
:NyquistVelocity-value = "53" ;
:vcp-unit = "dimensionless" ;
:vcp-value = "21" ;
:radarName-unit = "dimensionless" ;
:radarName-value = "KTLX" ;
:Elevation = 0.46875 ;
:ElevationUnits = "Degrees" ;
:MissingData = -99900. ;
:RangeFolded = -99901. ;
float NyquistVelocity(Azimuth) ;
NyquistVelocity:Units = "MetersPerSecond";
LatLonGrid netcdf format
The origin of the Lat-Lon-Grid is at the northwest corner (this is 0,0).dimensions:
Lat = 650 ;
Lon = 700 ;
variables:
float SHI(Lat, Lon) ;
SHI:Units = "dimensionless" ;
// global attributes:
:TypeName = "SHI" ;
:DataType = "LatLonGrid" ;
:Latitude = 37. ;
:Longitude = -100. ;
:Height = 1000. ;
:Time = 990402843 ;
:FractionalTime = 0.475000000005821 ;
:attributes = "" ;
:LatGridSpacing = 0.01 ;
:LonGridSpacing = 0.01 ;
:MissingData = -99900.f ;
:RangeFolded = -99901.f ;
SparseGrid netcdf format
If your data are sparse, you can save a whole bunch of space by saving only the non-missing data in a simple run-length encoded format. The global attributes and the other variables are the same as the regular formats. However, the dimensions are provided by pixel_x, pixel_y for the first two dimensions and run_length for the number of values in a run (default=1 if run_length is omitted). The original dimensions are also required even if no variables actually use them. The default background value is MISSING_DATA unless a different value is provided (see the SparseRadialSet example below).Example of a sparse lat-lon-grid:
dimensions:Example of a sparse RadialSet:
Lat = 650 ;
Lon = 700 ;
pixel = 23541 ;
variables:
float Reflectivity_0C(pixel) ;
Reflectivity_0C:Units = "dBZ" ;
short pixel_x(pixel) ;
short pixel_y(pixel) ;
int pixel_count(pixel) ;
// global attributes:
:TypeName = "Reflectivity_0C" ;
:DataType = "SparseLatLonGrid" ;
:Latitude = 37. ;
:Longitude = -100. ;
:Height = 0. ;
:Time = 990376569 ;
:FractionalTime = 0.584999999999127 ;
:attributes = " ColorMap Unit" ;
:ColorMap-unit = "dimensionless" ;
:ColorMap-value = "Reflectivity" ;
:Unit-unit = "dimensionless" ;
:Unit-value = "dBZ" ;
:LatGridSpacing = 0.01 ;
:LonGridSpacing = 0.01 ;
:MissingData = -99900.f ;
:RangeFolded = -99901.f ;
dimensions:
Azimuth = 360 ;
Gate = 460 ;
pixel = 4673 ;
variables:
float Azimuth(Azimuth) ;
Azimuth:Units = "Degrees" ;
float BeamWidth(Azimuth) ;
BeamWidth:Units = "Degrees" ;
float GateWidth(Azimuth) ;
GateWidth:Units = "Meters" ;
float PrecipConfidence(pixel) ;
PrecipConfidence:Units = "dimensionless" ;
short pixel_x(pixel) ;
short pixel_y(pixel) ;
int pixel_count(pixel) ;
// global attributes:
:TypeName = "PrecipConfidence" ;
:DataType = "SparseRadialSet" ;
:Latitude = 35.3330001831055 ;
:Longitude = -97.2779998779297 ;
:Height = 384. ;
:Time = 1122583396 ;
:FractionalTime = 0.947000000000116 ;
:attributes = " BackgroundValue ColorMap Unit" ;
:BackgroundValue-unit = "dimensionless" ;
:BackgroundValue-value = "0" ;
:ColorMap-unit = "dimensionless" ;
:ColorMap-value = "Fuzzy" ;
:Unit-unit = "dimensionless" ;
:Unit-value = "dimensionless" ;
:Elevation = 0. ;
:ElevationUnits = "Degrees" ;
:RangeToFirstGate = 0. ;
:RangeToFirstGateUnits = "Meters" ;
:MissingData = -99900.f ;
:RangeFolded = -99901.f ;
Example of a WindField netcdf format
This is essentially a 2 LatLonGrid in one file. Providing the meanwind is recommended, but not required.dimensions:
lat = 725 ;
lon = 775 ;
variables:
float uArray(lat, lon) ;
uArray:Units = "MetersPerSecond" ;
float vArray(lat, lon) ;
vArray:Units = "MetersPerSecond" ;
// global attributes:
:TypeName = "KMeansMotionEstimate" ;
:DataType = "WindField" ;
:Latitude = 37.5 ;
:Longitude = -101. ;
:Height = 0. ;
:Time = 1129321566 ;
:FractionalTime = 0.323000030519324 ;
:attributes = " Unit meanwind" ;
:Unit-unit = "dimensionless" ;
:Unit-value = "MetersPerSecond" ;
:meanwind-unit = "dimensionless" ;
:meanwind-value = "-0.00318577" ;
:LatGridSpacing = 0.00999999977648258 ;
:LonGridSpacing = 0.00999999977648258 ;
:MissingData = -99900.f ;
:RangeFolded = -99901.f ;
Tabular data
Tabular data are written out in XML. As with netcdf data, there are a number of attributes that require, to describe each Table as well as to describe each row.Each section is described below.
The entire table is enclosed in a datatable tag and has two child elements: datatype and data:
<datatable>
<datatype/>
<data/>
</datatable>
The datatype tag
The datatype tag describes the table as a whole. It has one stref tag, which describes the location and time of the table as a whole. It also has any number of attr tags, which refer to attributes of the table.<datatype name="MDA" >You can associate a column of values with a single attribute, although most commonly, you would associate only one value.
<stref>
<location>
<lat>
<angle units="Degrees" value="32.5731" />
</lat>
<lon>
<angle units="Degrees" value="-97.3031" />
</lon>
<ht>
<length value="228" units="Meters" />
</ht>
</location>
<time units="secondsSinceEpoch" value="799877001" />
</stref>
<attr name="ExpiryInterval" >
<datacolumn units="Minutes" name="ExpiryInterval" >
<item value="15" />
</datacolumn>
</attr>
</datatype>
The data tag
Tabular data consists of a number of columns. Each column would be described with subtag describing the value for each row:<data>The column named "AlgRank" is special -- it describes the relative importance of the rows, with 0 being the most important. It may be omitted, if all the rows are equally important.
<datacolumn units="dimensionless" name="AlgRank" >
<item value="0" />
<item value="1" />
<item value="2" />
<item value="3" />
</datacolumn>
<datacolumn units="Meters" name="Base" >
<item value="1438" />
<item value="2760" />
<item value="3760" />
<item value="2892" />
</datacolumn>
</data>
Contour Data in XML
The general format is as follows:<contours>
<datatype> ... </datatype>
<contourdata>
<array length="...">
<contour> ... </contour>
<contour> ... </contour>
...
</array>
</contourdata>
</contours>
The data type documentation is described above, in the section on Tabular data.
The contour is an array of locations; so a contour tag is organized as follows:
<contour>In the datatype element of the contour tag, you could provide an attribute named "Color"
<datatype> ... </datatype>
<locationdata>
<array length="...">
<location> ... </location>
</array>
</locationdata>
</contour>
so that different contours get different colors.