Platforms for WDSS-II
WDSS-II can be used as a research platform or to do real-time weather analysis. When used for research, you will typically use the same machine for running algorithms and for displaying the results. When doing real-time weather analysis, you will typically use an "algorithm" machine and view the generated products from a "display" machine. In that case, you will typically require a separate display machine for each real-time forecaster and enough algorithm machines to handle the ingest and algorithms for your domain (the larger the domain, the more machines you need).The minimum spec should be used when real-time performance is not critical. The regional-scale spec is usually the sweet spot in performance/cost. You could also farm off algorithm processing to a number of workstations, in which case, you can get away with older hardware.
I/O is the most critical aspect of algorithm machines, followed
closely by amount of RAM. The CPU specs are a distant third.
The graphics card is the most critical aspect for display machines --
almost nothing else matters.
So, if you have money left-over to upgrade your machine, try to
optimize the I/O first, memory next and CPU last. Be warned that
optimizing I/O invariably involves optimizing the read/write speeds not
search efficiency (1).
- Linux 2.4 (e.g: Red Hat Enterprise Workstations 3 or Fedora Core 1)
- Linux 2.6 (e.g: Red Hat Enterprise Workstation 4 or Fedora Core 2) kernels.
Hardware
Specifications
| Item | Minimum Spec for research version | Regional-scale (1000 km x 1000 km) algorithm machines | CONUS-scale
algorithm machines |
Display
-only machine for real-time applications |
| Processor | One Pentium IV or better | dual 32-bit Xeon or Athlon |
dual 64-bit Opteron or Xeon (2) |
single Xeon or better |
| Random Access Memory (RAM) | 512MB | 2GB per processor (ideally 4 GB per processor) |
8 GB per processor (ideally 16
GB per processor) |
2 GB |
| Storage (I/O) (3) |
1GB per data case | RAID 0 (dedicated RAID) -- 4 GB |
RAID 0 (dedicated RAID for each
machine) -- 20 GB |
1 GB |
| Video Card | 32MB graphics card supported by Linux | none |
none |
128MB NVidia GeForce4 for
display machine |
How many machines do you need?
These metrics for different scenarios may give you an idea:
- One regional-scale machine with 4 GB of RAM per processor can handle all the single-radar algorithms (ldm2netcdf, w2qcnn and w2circ) for 30 radars. You would need 3 to 4 such machines to do single-radar processing for the entire CONUS.
- One regional-scale machine with 4 GB of RAM can do all the
multi-sensor algorithms for the entire US.
- One regional-scale machine with 4 GB of RAM can do a nation-wide 2D reflectivity merger in real-time (2-minute update frequency).
- One CONUS-scale machine with 12 GB of RAM can do a nation-wide 3D reflectivity merger in real-time (5-minute update frequency).
- One regional-scale machine can do both single-radar and
multi-radar processes for a 1000km x 1000km domain.
Notes
- Don't fall for sales pitches that try to sell you storage area networks or other distributed storage solutions -- you need a dedicated RAID for each box with a Gigabyte connection going from the machine to the storage for optimal transfer speeds.
- Avoid 64-bit Itanium processors since gcc optimization on Itanium is poor; we've not been successful in porting our software to use Intel's compiler.
- Make sure that there is enough space in /tmp (at least 500 MB
free) since we use /tmp rather heavily. If you don't have space in
/tmp, set your TMPDIR environment variable to point to some place else.
Linux heavily optimizes the use of /tmp, and may not do so for other
directories.