IMPLEMENTATION ON THE NCEP CRAY
Roland R. Draxler
Office of Oceanic and Atmospheric Research
Air Resources Laboratory
1315 East West Hwy
Silver Spring, MD 20910
July 16, 1999
The HYSPLIT model for the prediction of dispersion and deposition of radiological pollutant
plumes was fully transferred to the NWS/NCEP Cray computer system on 26 October 1998 at
1200 UTC. The HYSPLIT model has been used as the dispersion model component of NCEP /
ARL's Regional Specialized Meteorological Center (RSMC) for transport model products since
the RSMC's inception in 1992. Prior to its transfer to NCEP, the model was run operationally on
ARL's computer which is not the most suitable environment for use in an operational 7/24
NOAA has specific statutory responsibility1 to respond to a variety of environmental
disturbances. Appropriate response to the majority of these incidents falls well within the
capability of the NOAA Line Offices (LOs). These offices are equipped to fulfill their respective
duties regarding the environment, public, state and other Federal agencies and foreign
A domestic nuclear emergency presents such an instance of large-scale response involving many
elements of NOAA. An atmospheric nuclear emergency is any event that results in the release or
potential release of hazardous radioactive materials into the atmosphere and in these situations
the NOAA Response Plan for Nuclear Emergencies1 specifies that there shall be a formally
recognized Point-of-Contact (POC) within NOAA -- the Senior Duty Meteorologist (SDM) at
NCEP. Once notified of a nuclear emergency, the SDM will immediately contact each LO and,
if required, generate appropriate meteorological and dispersion model products.
NOAA capabilities to respond to an international nuclear emergency have been organized since
1992 through the structure of the World Meteorological Organization's (WMO) Regional
Specialized Meteorological Centers (RSMC), operated jointly by the National Weather Service's
National Centers for Environmental Prediction (NWS/NCEP) and the Office of Oceanic and
Atmospheric Research's Air Resources Laboratory (OAR/ARL). A demonstration of NOAA's
and Canadian Meteorological Centre's (CMC) dispersion modeling capabilities was made to the
WMO's Commission for Basic Systems (CBS) during their Tenth Session. Following this
demonstration, RSMC designations for Washington and Montréal became effective 1 July 1993
with NOAA and CMC jointly providing support to National Meteorological Services in Regional
Associations (RA) III and IV (Central and South America)2. The SDM is the RSMC initial POC.
In the event of an accident, the SDM would perform the initial simulation of the radioactive
plume transport and deposition with the operational dispersion model (HYSPLIT4) on the
NCEP computer system using a previously agreed upon default scenario . After the initial
response by NCEP and consultation with ARL, the dispersion model simulation configuration
may be modified to more accurately reflect the conditions of the accident. All operational
emergency simulations would continue to be performed on NCEP's computer, monitored and
controlled by the SDM, with ARL providing HYSPLIT's input configuration data. Exploratory
simulations for scenario variations may concurrently be run on ARL's computer systems.
Modeling System Description
The Environmental Emergency Response (EER) system is based around the HYSPLIT
Atmospheric Transport Model (ATM) developed, over a number of years at NOAA's Air
Resources Laboratory3 and recently modified in collaboration with the Australian Bureau of
Meteorology4. The ATM is driven by meteorological fields output from the operational
Numerical Weather Prediction (NWP) systems. HYSPLIT4 (Hybrid Single-Particle Lagrangian
Integrated Trajectories, version 4) can model the atmospheric transport and dispersion of
pollutant plumes originating from a variety of sources (e.g. nuclear, volcanic, fire smoke,
chemical, and other pollutants). The "hybrid" part of the acronym refers to the use of both
movable 'Lagrangian' (for the advection and diffusion calculations) and fixed 'Eulerian' (for the
concentration calculations) modeling frames of reference within the system. The application of
the model to a specific pollutant is accomplished through an input CONTROL file that defines
all relevant pollutant parameters5.
The current operational configuration of the system makes use of another "hybrid" feature of
HYSPLIT4, which was not available in version 3, i.e. a mixed algorithm which considers puff
dispersion in the horizontal and particle dispersion in the vertical. On the release of a single puff
of pollutant from a source, the puff will be advected by the mean wind and will expand as a
result of diffusion processes in the turbulent atmosphere. In the system, the puff is allowed to
grow laterally to a certain size, after which it splits into several new puffs, each with their
respective fraction of the pollutant mass. These new puffs will, in turn, be subject to advection
and diffusion. The splitting of puffs could also occur vertically. However, in the operational
configuration, particle - rather than puff - dispersion has been chosen for the vertical calculations.
(In cases of strong atmospheric mixing, puff splitting in the vertical can result in too many puffs
The HYSPLIT4 system also includes a number of other processes for removing, adding to, or
changing the composition of the pollutant plume. Dry deposition is the transport of pollutant
gaseous or particulate species onto surfaces. In the system, a dry deposition velocity can be
defined explicitly or can be calculated using details about the nature of the surface. For particles,
gravitational settling, requiring estimates of particle shape, size and density, is another option. In
wet deposition, the pollutant is scavenged by the atmospheric hydro meteors and is thus delivered
to the earth's surface. The HYSPLIT4 system allows for both within-cloud ("washout") and
below-cloud ("rainout") scavenging. If the winds are sufficiently strong, and the pollutant is not
bound to the surface, then resuspension can also occur. In the case of nuclear incidents,
radioactive decay is incorporated.
The HYSPLIT4 system can be run in a purely trajectory, or advective, mode producing either
forward or backward trajectory plots at specified levels. Alternatively, it can be run in a
dispersion mode producing exposure (or concentration) and surface deposition charts integrated
over various time periods and layers. (It is noted that it is not possible to run the dispersion
calculation backwards, since dispersion is an irreversible process.) The nature of a source can be
defined according to its strength, height and size, and the duration of emission.
There are three basic products associated with the default EER suite: trajectories, air
concentrations, and pollutant deposition. All products are in chart form and are produced using
NCAR graphics. Example charts produced from a recent simulation on the Cray are shown in
the attached illustrations (Figs. 1, 2, and 3). The forward trajectory chart is from a single source
starting simultaneously at three different levels: 500, 1500, and 3000 m agl. The lateral, or
horizontal, depiction of trajectory paths are annotated by the times (in UTC) at 6-hourly intervals.
The vertical motion is displayed below the main charts with tick marks indicating the 6-hourly
intervals. Air concentrations as given in time-integrated units averaged over a 24 hour period,
while deposition is shown as the total amount accumulated at the ground over the entire
The meteorological component of the EER system can use NWP fields from either the Aviation
(AVN) run of the Global Spectral Model (GSM) or for higher spatial and temporal resolution, the
Regional Spectral Model (RSM), which can be relocated globally over any location as needed for
the dispersion simulation. The HYSPLIT model calculation requires a time series of gridded
fields such as surface pressure, surface height, precipitation and the multi-level fields of
temperature, humidity and wind components. These fields are pre-processed from the GRIB files,
interpolated horizontally, if not already on a conformal grid, and then packed into a form suitable
for direct input into HYSPLIT. No vertical interpolation is required because the model can
handle data fields on either pressure, sigma, height, or hybrid vertical coordinates. The
customized ATM meteorological data file is a fixed-record length, direct-access, platform
independent format, containing data from all time periods, which then permits either forward or
backward calculations through the meteorological data.
The default configuration of the current EER system would use the AVN fields for the initial
calculation based upon the location and time of the pollutant release. The AVN fields for the
most recent forecast hour available (00, 06, 12, or 18 UTC) are selected and the meteorological
pre-processor extracts data points mapped to a 100 x 100 element grid at 100 km resolution at 3
hour intervals from 48 hours prior to the forecast initialization time to the end of the 72 hour
If finer scale meteorological features require more detailed meteorological data (and computer
resources are available), the SDM has the option to initiate a run of the relocatable version of the
Regional Spectral Model (RSM). The RSM is cast as a perturbation upon the global model, and
thus makes use of not only the boundary conditions provided by the driving global model, but
also the interior solution. The RSM produces high resolution forecasts almost identical to those
that could be obtained by the global model running at very high equivalent uniform resolution.
The EER version of the RSM was modified to permit it to be more easily relocated on-demand
over any point on the globe. The resolution is 40 km on a 145 x 144 x 28 level grid interpolated
to a polar stereo-graphic projection above 45 degrees latitude and a Mercator projection over the
equatorial latitudes. The model runs quickly because the higher resolution RSM fields are only
generated for the first 24 hours of transport. The dispersion calculation starts with the RSM data,
and then after the pollutant is mixed over a larger spatial domain, the calculation is automatically
carried over to use the AVN output fields.
Although the EER system is configured to run only with the AVN, RSM, or both, meteorological
pre-processing programs are available for the Eta and RUC models (as well as the ECMWF and
the Navy's COAMPS model) and can be implemented in the future if needed. Further the
HYSPLIT meteorological data format is identical to the data format used for VAFTAD, which is
currently produced operationally at a coarser resolution than is available with the HYSPLIT
The HYSPLIT model's development has always been done in conjunction with experimental
data for verification. Version 3 of the model was tested extensively with data from tracer field
experiments such as CAPTEX6 and ANATEX7,8 (Cross-Appalachian and Across North America
Tracer Experiments) with concomitant journal publications. The current EER and most recent
Version 4, was compared to measured balloon trajectories9, the ANATEX air concentration data,
measured deposition from the Chernobyl accident, and satellite photos from volcanic eruptions10.
HYSPLIT4 has been independently evaluated (not by the model's developer) to the air concentration results from the recent European Tracer Experiment (ETEX) and also compared to 27 other operational ATM models. There were two phases to the evaluation. In the real-time phase11 all ATM's used their own forecast NWP models to drive the ATM calculation to forecast the tracer concentrations prior to experiment (hence no model tuning was possible). In the post-experiment phase12, the measured air concentration data were provided to the modelers and all models were expected to use the same meteorological fields from the ECMWF. In both these evaluations, HYSPLIT results generally were in the middle of the performance range, in the real-time phase due to coarser resolution of AVN forecasts over Europe compared with the data fields available to the European NWP centers and in the post-experiment phase due to pre-processing the ECMWF data at a resolution to be consistent with the previous real-time simulation13.
In addition to model verification using experimental data, the EER system is tested monthly by
comparing our ATM results with those of the CMC, each of us using our own ATM and NWP
models. These tests are used to reinforce the cooperation between RSMCs Washington and
Montréal to respond jointly to emergencies. These regular monthly tests have been conducted
with the CMC since the fall of 1993 and the vast majority of the simulations have shown
comparable transport directions and air concentrations2. Largest differences between the ATMs
occur during events with substantial wet removal, with HYSPLIT showing much larger
deposition than the CMC model due to differences in the parameterization of the pollutant's wet
NCEP provides the 7/24 initial contact point for radiological assistance requests. In the event of
an accident, the SDM connects to dedicated dispersion models operational on the NCEP
computer system by invoking "sh setup_hysplit" from a CRAY 3 window. Details of the source
such as the latitude/longitude and the height (above ground level), actual time of release and
release amount per hour (if known) are entered in the script that generates the HYSPLIT control
file and submits the operational job.
After the completion of the job two graphical product output files will have been created, an
NCAR graphics gmeta file and the Postscript equivalent of the gmeta file. Each output file
contains 6 frames: a text cover page summarizing the simulation parameters, the trajectory
graphic, average air concentration maps from the analysis time to +24 h, +24 to +48 h, and +48
to +72 h, and a total deposition map for the period from the start of the release to +72 h. The
Postscript file is copied to the SDM workstation and locally printed. Normal product distribution
is by telephone facsimile. The Postscript file can be uploaded, either directly by telephone or
through the Internet to a commercial fax redistribution service where various lists of fax numbers
are maintained for product distribution to NMS of RAs III and IV, as well as WMO, and other
national and international organizations. The gmeta file is also automatically copied to the ARL
workstation, where depending upon the nature and sensitivity of the incident, the product may be
posted on the RSMC web page.
ARL is responsible for continued research, development, and model verification, including
implementing upgrades at NCEP. Although the principal development of the model was done at
ARL, several collaborators have made and will continue to make substantial contributions to its
further development. An operational version of the model is currently being run at the Australian
Bureau of Meteorology14, South African Weather Bureau (linked with their version of the Eta
model), Meteorological Services in the Czech Republic and Croatia. In addition the model is
installed at several universities (MIT, Hawaii, Michigan, Calabria, Rutgers, Toulouse) and
government agencies (NIST, US Air Force, Oak Ridge National Laboratory) and international
organizations (British Atmospheric Data Centre, Comprehensive Test Ban Treaty Organization).
NCEP is responsible for maintaining the HYSPLIT model in a 7/24 operational environment on
its computer system. NCEP also runs the GSM four times per-day and maintains the RSM for
application as needed. The GSM and HYSPLIT models reside in normal production suites where
both production and communication are fully supported by a staff of programmers and
The HYSPLIT model was officially implemented on the Cray at 12Z on Monday 26 October,
1998 with the following JIFs:
1154. SCRIPT Invoked by the SDM to control setup and submission.
1155. AVN2ARL Converts AVN GRIB file into HYSPLIT compatible format.
1156. RSM2ARL Converts Regional Spectral Model GRIB file to HYSPLIT format
1157. HYMODELT Trajectory model that creates ASCII end-point output file.
1158. TRAJRSMC Converts ASCII end-points file to NCAR graphics gmeta.
1159. HYMODELC Concentration and deposition model creates binary gridded output file.
1160. CONCRSMC Converts binary concentration file to NCAR graphics gmeta.
Future Model Developments
There are a variety of potential changes in various stages of research and development. Because
the model was developed on an IBM AIX platform, no problems are anticipated upgrading the
EER system from the CRAY to the SP. Currently being tested are faster algorithms to improve
HYSPLIT's processing of gridded meteorological data and the meteorological pre-processor will
be upgraded to handle more vertical levels. New graphical display programs are being evaluated
that will directly produce the Postscript graphical output files using public domain Fortran
callable subroutines, therefore eliminating the need for NCAR graphics. Future research plans
call for the development of a parallelized version of HYSPLIT that can be used to generate an
ensemble prediction (i.e. the probability that a certain concentration level will be exceeded) and
the integration of the dispersion model component directly into one of the NWP models such as
Although currently the radiological and volcanic ash EER systems are independent, ARL is in the
process of modifying the VAFTAD program to use all the same subroutine libraries as
HYSPLIT, thereby reducing the need to support various programs that all do variations of the
same task. The HYSPLIT updated VAFTAD will be able to use all the same meteorological data
files. In addition a variation of the VAFTAD display program is under development that will
also be able to use the same public-domain Postscript libraries as HYSPLIT.
Internal Users and Customers
The Federal Radiological Emergency Response Plan (FRERP - 1 May 1996) outlines contributions that Federal Agencies can make in the event of a domestic nuclear emergency.
Any Lead Federal Agency (LFA such as the Environmental Protection Agency and the Nuclear
Regulatory Commission) may request NOAA assistance in the event of a nuclear emergency. In
addition, Appendix C of the FRERP specifies roles and capabilities of the Department of
Commerce, of which the following is attributed to reside within NOAA: Preparing and
disseminating predictions of plume trajectories, dispersion, and deposition of radiological
material released into the atmosphere. In addition to supporting other Federal Agencies, there
are requirements within NOAA to maintain a steady flow of reliable information (I) to the Under
Secretary/Administrator, (ii) to those line offices with specific responsibilities, and (iii) to NWS
forecast offices and other NOAA organizations directly involved with supporting management of
the actual on-site response effort.
The way in which NOAA responds to an international nuclear incident is already well prescribed
by WMO in the related RSMC documentation in that NOAA provides guidance to the National
Meteorological Services of countries in RA III and IV if they request our assistance. In addition
we provide backup to the CMC and the Australian BoM (for RA V) in the event they are unable
to provide services. The precise mechanism for interactions with other Federal Agencies that are
also involved, through direct association with the International Atomic Energy Agency (IAEA)
for example, is the topic of ongoing debate among the agencies concerned.
Customer Benefits of Implementation
The existing version of the EER modeling system resides on an ARL workstation.
Implementation on NCEP's Cray provides 7/24 operation, increased reliability, and faster
computational speeds. Due to computer resource limitations at ARL, global AVN
meteorological data files are maintained at 200 km spatial and 6 h temporal resolution.
Implementation on the Cray provides computations at 100 km spatial and 3 h temporal
resolution and in addition provides a global high resolution capability at 40 km using the RSM
model. In addition, when the entire EER system resides on the CRAY, both the meteorological
and dispersion programs will be on the same platform, reducing the potential failures in the
system due to communication problems.
1) National Oceanic and Atmospheric Administration Response Plan for Nuclear Emergencies,
approved and signed by the Deputy Under Secretary, 13 Feb 1997.
2) Draxler, R.R., M. Jean, B. Hicks, and D. Randerson, 1997. Emergency preparedness --
Regional Specialized Meteorological Centers at Washington and Montreal. Radiation Protection
Dosimetry, 73: 27-30.
3) Draxler, R.R. 1992. Hybrid single-particle Lagrangian integrated trajectories (HY-SPLIT):
Version 3.0 -- User's guide and model description. NOAA Tech. Memo. ERL ARL-195, 26 pp.
4) Draxler, R.R., and G.D. Hess, 1997. Description of the HYSPLIT_4 modeling system. NOAA
Technical Memo ERL ARL-224, December, 24 p.
5) Draxler, R.R., 1999. The HYSPLIT_4 User's Guide, NOAA Technical Memo, in preparation.
6) Draxler, R.R. and Stunder, B.J.B. 1988. Modeling the CAPTEX vertical tracer concentration
profiles, J. Appl. Meteorol., 27, 617-625.
7) Draxler, R.R., Dietz, R., Lagomarsino, R.J. and Start, G. 1991, Across North America
Tracer Experiment (ANATEX): Sampling and Analysis, Atmos. Environ., 25A, 2815-2836.
8) Draxler, R.R. 1991. The accuracy of trajectories during ANATEX calculated using dynamic
model analyses versus rawinsonde observations, J. Appl. Meteorol., 30, 1446- 1467.
9) Draxler, R.R., 1996. Trajectory optimization for balloon flight tracking. Weather and
Forecasting 11: 111-114.
10) Draxler, R.R., and G.D. Hess, 1998. An overview of the HYSPLIT_4 modelling system for
trajectories, dispersion, and deposition. Australian Meteorological Magazine, 295-308.
11) Graziani, G., Klug, W., Mosca, S. 1998. Real-Time Long-Range Dispersion Model
Evaluation of the ETEX First Release, European Commission Joint Research Centre, EUR
17754 EN, 213p.
12) Mosca, S., Bianconi, R., Bellasio, R., Graziani, G., Klug, W., 1998. ATMES II - Evaluation
of Long-Range Dispersion Models Using Data of the 1st ETEX Release, European Commission
Joint Research Centre, EUR 17756 EN, 455p.
13) Hess, G. D., G.A. Mills, and R.R. Draxler, 1997. Comparison of HYSPLIT_4 model
simulations of the ETEX data, using meteorological input data of differing spatial and temporal
resolution. ETEX Symposium on Long-Range Atmospheric Transport, Model Verification and
Emergency Response, 13-16 May, Vienna, Austria
14) Stuart, P. 1998, HYSPLIT4 Atmospheric Transport Model, Analysis and Prediction
Operations Bulletin No.44, 22 October 1998, Australian Bureau of Meteorology, Melbourne
Figure 1. Example output from the trajectory calculation
showing 72 h forecast trajectories started at 500, 1500, and
3000 m agl at 1800 UTC on 2 June 1999.
Figure 2. Example output from the first air concentration
chart, showing the time integrated air concentrations
between two synoptic times (1200 UTC 2 June through
1200 UTC 3 June) nearest to the pollutant release time.
Figure 3. Example output from the deposition chart, showing the total deposition from the start of the release through 1200 UTC 5 June.