Warning Decision Support System -- Integrated Information

Automated algorithms, tools and displays for analysis, diagnosis and forecasts of severe weather phenomena       www.wdssii.org

WDSS-II

Real-time Data

Discussion Forum
WDSS-II Blog

Documentation
For developers
Obtaining WDSS-II

Contact


Real-time monitoring of WDSS-II processes

WDSS-II algorithms can be started individually on the command-line, but in a real-time environment, you'd typically want to set up multiple processes and link them together (so that the output of one process feeds into the input of another process). You'd also want to start and stop processes as a group (for example, all the processes depending upon data from the radar KAMA). To do that, you would use the "w2alg" capability.

Installation

  1. Download and untar the w2algrun.tgz file somewhere. You'd want to keep this in a directory that gets periodically backed up.
    • cd $HOME
    • tar xvfz /tmp/w2algrun.tgz
  2. cd into the w2algrun directory
    • cd w2algrun
  3. run ./autogen.sh
    • ./autogen.sh
  4. Create a w2alg.conf. See the example at the end of this webpage for instructions on how to do this.
  5. Softlink the w2alg in that directory to some directory that's already in your PATH.
    • cd ~/WDSS2/bin
    • ln -s ~/w2algrun/w2alg .

LDM Configuration

WDSS-II algorithms need to ingest data from remote sensors. The data need to make it to the workstation on which WDSS-II algorithms are running. Such data are typically transmitted through Unidata's LDM software. For more information on LDM, please consult your LDM user manual.

Here are example entries that would go into LDM's pqact.conf for some common meteorological sources. Make sure to request the data in your ldmd.conf.
Data Source
Name of file
Entry
WSR-88D (NWS radar network)
pqact.conf
NEXRD2  ^L2-([^/]*)/(....)/([0-9][0-9][0-9][0-9][0-1][0-9][0-3][0-9][0-2][0-9][0-5][0-9][0-9][0-9])
        PIPE    /home/ldm/bin/nexradII_bz       /home/ldm/data/nexradII/\2/\3.raw

TDWR (FAA weather radar)
pqact.conf
EXP     ([^/]*)/(OKC)([0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9](-)[0-9][0-9][0-9][0-9][0-9][0-9](-)[0-9]+(.)[0-9][0-9](-)[0-9]+(-)[0-9]+(-)[0-9]+(.)(bz2))
        FILE    /data/realtime/tdwr/Raw\2/\2\3

RUC (FSL Rapid Update Model)
pqact.conf
NMC2   [^/]*/RUC2/#252/([0-9][0-9][0-9][0-9][0-1][0-9][0-3][0-9][0-2][0-9][0-5][0-9])/F000/([A-Z]*)/([^/]*)(!) ([0-9][0-9][0-9][0-9][0-9][0-9])
PIPE   /home/wdssii/WDSS2/bin/gribToNetcdf -o /data/realtime/model/ruc2 -P



Quick Start

w2alg launches each algorithm process and sets it up with a time-stamped, rotating log and in an environment such that if the algorithm dies, it is restarted automatically.

w2alg reads w2alg.conf and uses that information to determine what processes to start and with what command-line parameters.  These are some of the most useful commands:

w2alg start all
Starts all the algorithms
w2alg start ktlx
Starts all the algorithms in the group KTLX
w2alg start ktlx a2
Starts the a2 algorithms in the group KTLX
w2alg stop all Stops all the algorithms
w2alg stop ktlx Stops all the algorithms in the group KTLX
w2alg status kltx qcnn
Provides status of the qc algorithm on KTLX
w2alg status kltx
Provides status of all the KTLX algorithms.
w2alg tail kltx qcnn
Tails the log of the qc algorithm on KTLX
w2alg tail mainlog
Tails the mainlog which tells you which algorithms died and were restarted.

Typically, every data source (radar, satellite, etc.) is a group, and every algorithm in it is a manager. For every algorithm, you would set the following parameters:
In the command-line parameters, you might find it useful to use globally defined variables (defined at the top of the w2alg.conf file -- $M1 in the example below defines a earth-centric domain specifying -t, -b and -s for all the algorithms that need them. See w2merger for example) and variables such as $GROUP.

Note that even though this example, for simplicity, directly includes the command-line parameters, you can avoid repetitiveness by starting the algorithms with a --configFile=something.xml and pass the command-line options using the configFile. This way, you can reuse command-line parameters across groups.


Example w2alg.conf file (for a regional WDSS-II domain)


# group GLOBAL sets for all groups
# Any 'set' overrides within hierarchy
#
group GLOBAL
                                                                               
set M1 " -t "40 -82 20" -b "32 -73 0" -s "0.01 0.01 1" "
                                                                               
# Default W2Logger parameters per algorithm (can be overridden below)
set LOG_SIZE "25000"  # Size of log file in -s
set NUM_LOGS "3"      # Max number of logs
set LOG_DIR "/home/wdssii/logs"
                                                                               
# Default W2Run parameters
set W2RUN_LOG_SIZE "25000"  # Log for shared w2Runners (in bytes) monitoring alg processes
set W2RUN_NUM_LOGS "10"
set W2RUN_LOG_DIR "/home/wdssii/logs"
set RESTART_SLEEP ".5"       # Time to sleep (in secs) before restarting dead process
set PROCESS_DWELL "2"       # Min time between starting processes
                                                                               
# Default ProcWatcher parameters
set PROC_WATCH_USE "1"      # Use proc watch if non-zero
set PROC_WATCH_SLEEP "300"  # 300 seconds between proc logs
 
manager RSSD      # RSSD support module
 
#group all
#   manager PROCESS-RSSD      # RSSD support module
#     set PROCESS_EXE "rssd"
#     set PROCESS_PARAMS "-t"
#
# 'w2alg start global top'
#
#manager TOP                 # Open a gtop window using the TOP module
 
#
# Example of using the LOG.pm module
# Run this log by "w2alg start monitor w2runlog"
#group MONITOR
#  manager LOG-W2RUNLOG     # Create log watcher for w2run1.log
#    set LOG_FILE "w2run1.log"
#    set LOG_NAME "W2Run Log"
 
# Example of using the PROCESS.pm module
# Run this by "w2alg start ktlx clock1"
# Run this by "w2alg start ktlx clock2"
#group KRAX                   # Experimental group KRAX
#  manager PROCESS-CLOCK1     # Test PROCESS module with unkillable xclock :)
#    set PROCESS_EXE "xclock"
#    set PROCESS_PARAMS "-digital"
#  manager PROCESS-CLOCK2     # Test PROCESS module with unkillable oclock :)
#    set PROCESS_EXE "oclock"
 
# Multigroup example:
# can do "w2alg start test xterm1" or "w2alg start test2 xterm1"
# each will use the $GROUP macro to set title correctly
#group TEST, TEST2
#   manager PROCESS-XTERM1
#     set PROCESS_EXE "xterm"
#     set PROCESS_PARAMS "-t "$GROUP""
 
group CLOUD
 
   manager PROCESS-CLOUDCOVER
     set PROCESS_EXE "/home/wdssii/WDSS2/bin/w2cloudcover"
     set PROCESS_PARAMS "-i "xmllb:cairo:/data/realtime/satellite/code_index.lb xmllb:wrwdss:/data/ruc2/products/code_index.lb" -o /data/realtime -l /data/realtime/radar/multi/code_index.lb -S TMP-HTGL:20km2_analysis -r -e 5  $M1 -I --verbose"
 
group LTG2D
 
   manager PROCESS-LTG2D
     set PROCESS_EXE "/home/wdssii/WDSS2/bin/w2ltg"
     set PROCESS_PARAMS "-i "xmllb:cairo:/data/realtime/ltng/ltg_index.lb xmllb:/data/realtime/radar/multi/code_index.lb" -o /data/realtime/ltg -t "40 -82" -b "32 -73" -s " 0.01 0.01"  -e 15 -p "15 30" -I "Reflectivity_0C Relfectivity_-20C" -r -f 30 --verbose"
 
 
#group KCLX,KLTX,KMHX,KRAX,KAKQ,KLWX,KFCX
group KMHX,KRAX,KAKQ,KLWX,KFCX
 
   manager PROCESS-A2
     set PROCESS_EXE "ldm2netcdf"
     set PROCESS_PARAMS "-i /home/ldm/data/nexradII/$GROUP -s $GROUP -o "/data/realtime/radar/$GROUP /data/realtime/radar/U-$GROUP" -k -D "
 
   manager PROCESS-QCNN
     set PROCESS_EXE "w2qcnn"
     set PROCESS_PARAMS "-i xmllb:/data/realtime/radar/U-$GROUP/code_index.lb -o /data/realtime/radar/$GROUP -r -u"
 
   manager PROCESS-LLSD
     # OVERRIDE Default W2Logger parameters
     set LOG_SIZE "100000"  # Size of log file in -s
     set NUM_LOGS "10"      # Max number of logs
     set LOG_DIR "/home/wdssii/logs"
     set PROCESS_EXE "llsd"
     set PROCESS_PARAMS "-i xmllb:/data/realtime/radar/U-$GROUP/code_index.lb -o /data/realtime/radar/$GROUP -t -v -r "
 
   manager PROCESS-SCIT2D
     set PROCESS_EXE "swatScit2D"
     set PROCESS_PARAMS "-i xmllb:/data/realtime/radar/U-$GROUP/code_index.lb -o /data/realtime/radar/$GROUP -f -r -s $GROUP"
     set DIR "/home/wdssii/algconfig/$GROUP/scit"
 
group MULTI
   manager PROCESS-MERGER
     set PROCESS_EXE "w2merger"
     set PROCESS_PARAMS "-i "xmllb:/data/realtime/radar/KLWX/code_index.lb xmllb:/data/realtime/radar/KFCX/code_indexl.b xmllb:/data/realtime/radar/KAKQ/code_index.lb xmllb:/data/realtime/radar/KRAX/code_index.lb xmllb:/data/realtime/radar/KMHX/code_index.lb xmllb:unioncity:/tmp/NSE/code_index.lb" $M1 -p "0.5" -o /data/realtime/radar/multi -c -v -e 60 -r -H -C 5 -v -V -M xmllb:/data/realtime/radar/multi/code_index.lb -I ReflectivityQC -T 6 --verbose"
#     set PROCESS_PARAMS "-i "xmllb:/data/realtime/radar/KCLX/code_index.lb xmllb:/data/realtime/radar/KAKQ/code_index.lb xmllb:/data/realtime/radar/KLTX/code_index.lb xmllb:/data/realtime/radar/KRAX/code_index.lb xmllb:/data/realtime/radar/KMHX/code_index.lb xmllb:/data/realtime/radar/KLWX/code_index.lb xmllb:/data/realtime/radar/KFCX/code_index.lb xmllb:unioncity:/tmp/NSE/code_index.lb" $M1 -p "0.5" -o /data/realtime/radar/multi -c -v -e 60 -r -H -C 5 -v -V -M xmllb:/data/realtime/radar/multi/code_index.lb -I ReflectivityQC"
 
   manager PROCESS-MERGELLSD
     set PROCESS_EXE "w2merger"
     set PROCESS_PARAMS "-i "xmllb:/data/realtime/radar/KCLX/code_index.lb  xmllb:/data/realtime/radar/KMHX/code_index.lb xmllb:/data/realtime/radar/KRAX/code_index.lb xmllb:/data/realtime/radar/KAKQ/code_index.lb xmllb:/data/realtime/radar/KLWX/code_index.lb  xmllb:/data/realtime/radar/KFCX/code_index.lb " $M1 -p "0.001" -I "AzShear" -o /data/realtime/radar/multi -c -e 60 -r -C 1 -V "
#     set PROCESS_PARAMS "-i "xmllb:/data/realtime/radar/KLTX/code_index.lb xmllb:/data/realtime/radar/KCLX/code_index.lb  xmllb:/data/realtime/radar/KMHX/code_index.lb xmllb:/data/realtime/radar/KRAX/code_index.lb xmllb:/data/realtime/radar/KAKQ/code_index.lb xmllb:/data/realtime/radar/KLWX/code_index.lb  xmllb:/data/realtime/radar/KFCX/code_index.lb " $M1 -p "0.001" -I "AzShear" -o /data/realtime/radar/multi -c -e 60 -r -C 1 -V "
 
#   manager PROCESS-QCINFO
#     set PROCESS_EXE "qcinfo"
#     set PROCESS_PARAMS "-i "xmllb:/data/realtime/radar/U-KLTX/code_index.lb xmllb:/data/realtime/radar/U-KCLX/code_index.lb xmllb:/data/realtime/radar/U-KMHX/code_index.lb xmllb:/data/realtime/radar/U-KRAX/code_index.lb xmllb:/data/realtime/radar/U-KAKQ/code_index.lb" -s "KLTX KAMA KMHX KRAX KAKQ  " -o /data/realtime/radar/multi -r"
 
 
   manager PROCESS-SEGMOTIONLL
     set PROCESS_EXE "w2segmotionll"
     set PROCESS_PARAMS "-i "xmllb:/data/realtime/radar/multi/code_index.lb" -o /data/realtime/radar/multi -T MergedReflectivityQCComposite -r  -t "30 60" -d "20 60" -g -E 10 -O 5"
 
   manager PROCESS-MRSCIT
     set PROCESS_EXE "scit3D"
     set PROCESS_PARAMS "-i "xmllb:/data/realtime/radar/KCLX/code_index.lb xmllb:/data/realtime/radar/KMHX/code_index.lb xmllb:/data/realtime/radar/KRAX/code_index.lb xmllb:/data/realtime/radar/KLWX/code_index.lb  xmllb:/data/realtime/radar/KFCX/code_index.lb xmllb:/data/realtime/radar/KAKQ/code_index.lb xmllb:unioncity:/tmp/NSE/code_index.lb" -o /data/realtime/radar/multi -r "
#     set PROCESS_PARAMS "-i "xmllb:/data/realtime/radar/KLTX/code_index.lb xmllb:/data/realtime/radar/KCLX/code_index.lb xmllb:/data/realtime/radar/KMHX/code_index.lb xmllb:/data/realtime/radar/KRAX/code_index.lb xmllb:/data/realtime/radar/KLWX/code_index.lb  xmllb:/data/realtime/radar/KFCX/code_index.lb xmllb:/data/realtime/radar/KAKQ/code_index.lb xmllb:unioncity:/tmp/NSE/code_index.lb" -o /data/realtime/radar/multi -r "