Skip Navigation Linkweather.gov 
NOAA logo - Select to go to the NOAA homepage National Weather Service

 Headquarters Office NWS logo
NWS Office of the Chief Information Officer/Telecommunication Operations Center
 
.
Text Forecast
 
 
 
USA.gov is the U.S. government's official web portal to all federal, state and local government web resources and services.

 
Help Guide

FTP Retrieval of Data Products
using
Observational Data as Examples

Directory and Filename Structure for Anonymous FTP

There are two different systems used by our file servers in constructing our directory names and file names. One system is an "English descriptive format" used for individually stored forecasts and observations. The other system is based upon a File Name Standard. Files of "WMO bulletins" (note 1) use the Telecommunication Operations Center "File Name Standard" format for storage of meteorological information. There are, therefore, two types of file content formats available on the servers. One is a file of grouped WMO messages available as received from various source communication lines. The other type is files of products or observations that do not contain a WMO abbreviated heading within the file. The second type were either never transmitted as WMO bulletins or were parsed from the original bulletins. The WMO bulletins are structured items for transmission and contain observational data and forecast products. Products which do not use the WMO bulletin format structure but do use the WMO GRIB code for providing the forecast information are also stored under the "File Name Standard". They normally contain large forecast model computer forecast grid field products generated by the NWS/NCEP. The documentation in the access procedures for FTP/HTTP Services web page gives an overview of the data and products available at the Gateway that use this standard. The user must understand the meteorological code formats to be able to decode and display all coded data and forecast products.

The observation files (or bulletin (note 1) collectives of observations) are transmitted, received and stored under the TOC File Name Standard. These files contain data in many standard WMO meteorological code forms depending upon the type of data report. The observations are available "as received" and come from many world-wide sources. You are referred to the Observed Data (Data-Specific Collective Files) web page for details on all data-specific types available on our servers. The WMO collectives of METAR reports are included in these files as a data type under the "TOC File Name Standard". All of the model forecast products are stored under the "TOC File Name Standard" for FTP retrieval. The individual aviation METAR observation and Terminal Area Forecast (called TAF) are also stored under the "English Descriptive Format" in a "parsed" file set for site selective individual retrieval.

There are "Parsed" National Weather Service Forecast Office WARNINGS & FORECASTS stored on the server as well. These products can be found the same way as observations. The file tree is different beyond the /data/ sub-directory. The server user should review the explaination of the various directory trees for a better understanding of how to locate the warnings and forecasts they desire. Remember that the last received product is the only one available and that the warnings and forecasts are file names using forecast "zone" numbers by state. We do not store old data on the server in the parsed files. The URL is http://weather.noaa.gov/pub/data/ for reviewing the directory tree with a web browser and under the URL ftp://weather.noaa.gov/data/ to review the file structures using FTP through the windows browser or by using the standard windows FTP application software.

Retrieval of Selected Observations

This is a "HOW TO" help guide. It may help to establish a process to pull selected data (such as exampled here for METAR reports) from our "English descriptive format" server resources. You can also pull all of our METAR bulletin files from our server resources using the "TOC File Name Standard" as illustrated later on this page. These procedures focus on retrieving individual observations which were generated in the METAR code. However, the same method can be used for retrieval of individual text forecasts at the Gateway. The aviation data file names are "ICAO Site Identifiers" for the METAR reports and the TAFs. Other information, such as the English language text forecast, is individually stored under NWS zone forecast identifier numbers or with other meaningful descriptive file names. These files contain only the latest available observation, warning, or forecast received at the Gateway. Old observations and forecasts are not available at the Gateway under the "English descriptive format".

Recommended Methods for Retrieval

Several recommended methods are described in the following procedures. The METAR is an hourly observation in the United States and in many other countries around the world. Some countries take METAR observations on a half hourly basis. Constant FTP retrievals to our servers for the latest observation results in service access denial for others. The design of your retrieval processes must consider availability of the data. The METAR data is best requested between 10 and 15 minutes past the hour to account for central processing time. See our page on Individual Reports and Cycle Files for a discussion on vailibility.

Decoded METAR Reports

Here is an example on the retrieval of a METAR observation for a selected location. The process uses the standard anonymous FTP protocol. For multiple reporting locations you should use the "mget" multiple retrieval FTP procedure. It is strongly recommended not to open multiple FTP sessions beyond three (3) to five (5) for retieving data.

1. Use this URL address = ftp://tgftp.nws.noaa.gov/data/observations/metar/decoded/ to establish your list of sites to retrieve (you must know the 4-letter site dentification)

2. You will be returned a file list (note2) of "4-letter site ID" files with names ending with ".TXT" to select from [be aware that the server is character case sensitive]

3. Establish a list of filenames, by station name (ICAO site Identifier) to use in the FTP, and use the following format of the FTP call in your retrieval program

4. example: URL = ftp://tgftp.nws.noaa.gov/data/observations/metar/decoded/KIAD.TXT for KIAD = Dulles International Airport
(note the site filename is in upper case)

5. You will get, for example, a returned file with a content in this form in ASCII for a decoded METAR from KIAD

Washington DC, Washington-Dulles International Airport, VA, United States (KIAD)
38-56-05N 077-26-51W 93M
Jan 23, 2001 - 11:51 AM EST / 2001.01.23 1651 UTC
Wind: Variable at 3 MPH (3 KT):0
Visibility: 8 mile(s):0
Sky conditions: mostly clear
Temperature: 30.0 F (-1.1)
Dew Point: 21.0 F (-6.1 C)
Relative Humidity: 68%
Pressure (altimeter): 30.26 in. Hg (1024 hPa)
ob: KIAD 231651Z VRB03KT 8SM FEW200 M01/M06 A3026 RMK AO2 SLP252 T10111061
cycle: 17

Coded METAR Reports

1. Use this URL address = ftp://tgftp.nws.noaa.gov/data/observations/metar/stations/ and again you must know the 4-letter site identification

2. You will be returned a file list (note2) of 4-letter site ID files to select from (be aware that the server is case sensitive and note the use of capital letters in the filenames)

3. Establish a list of FTP calls with specific filenames, by station name {site Identifier} to FTP with,

4. Use this format of the URL = ftp://tgftp.nws.noaa.gov/data/observations/metar/stations/EGLL.TXT for EGLL = London / Heathrow Airport

5. You will be returned, for example, a file with a content in this form in ASCII for the latest Heathrow Airport METAR report.

2001/01/24 18:20
EGLL 241820Z 16009KT 9999 SCT042 BKN160 08/04 Q0995 TEMPO 5000
SHRA BKN015

6. The METAR code is found in the WMO manual 306 "Manual on Codes" Volume I Part A and can be ordered through the American Meteorological Society. A brief explaination of the code is available as NOAA/PA 96052 issued in March 1996. [see front and back of NOAA/PA 96052]

Coded METAR Collectives

In this example you will be using the server resources that use the "TOC File Name Standard" system for storing data. This storing method allows for data to be available for up to 24 hours after being received at the Gateway. You can not be site specific and must retrieve all of the METAR observations received at the Gateway for the time period selected in your retrieval.

1. Use this URL address ftp://tgftp.nws.noaa.gov/SL.us008001/DF.an/DC.sflnd/DS.metar/

2. You will be returned a file list (note2) of files containing all METAR bulletins (note 1) received at the Gateway. Each file contains data receipt over five (5) minute time period intervals (be aware again that the server is case sensitive and saves the last sequence number in the file you last read)

3. Then you can establish a list of filenames, by "sn.XXX {sequence number} to pull the files with, and use multiple FTPs of this form

4. URL = ftp://tgftp.nws.noaa.gov/SL.us008001/DF.an/DC.sflnd/DS.metar/sn.0020 (an example to read file "0020 in the METAR sub-directory). For the range of files you want, you must establish a set of files to read from the last read sequence number of the previous read to the last written number you find in the file list just read. For example if the last time you ended your read with the sn.0020 file the next read will start with the sn.0021 file. A group of multiple FTP calls would have sn.0021, sn.0022, sn.0023, and so on up to and including the last number found updated in the file list just check to see if more data was available to be pulled.(note2)

5. You will get a returned set of files each of which would have contents in ACSII. An example of the content of a file read would have the format illustrated below.

This was the contents of a file named "sn.0131" on January 25, 2001 written at 10:05 UTC

####018000110####
SAUS43 KDDC 250957
MTRDDC
METAR KDDC 250956Z 16011KT 10SM CLR M03/M08 A3034 RMK AO2 SLP288
T10331078
####018000120####
SAFI31 EFHK 250950 COR
METAR
EFOU 250956Z 15008KT 120V190 3000 -SN SCT018 SCT023 BKN040 M04/M06
Q1001 1239//45=
####018000514####
SANO32 ENMI 250950
METAR
ENAL 250950Z 21014KT 9999 FEW035TCU SCT045 07/01 Q0989=
ENCN 250950Z 17011KT 150V210 3000 -SHRA BR BKN003 05/04 Q0997=
ENHD 250950Z NIL=
ENRO 250950Z VRB01KT 9999 FEW015 SCT060 BKB200 02/M03 Q0991=
ENRY 250950Z 20008KT 170V230 5000 BR FEW003 SCT005 BKN008 05/04 Q0998
      TEMPO 2000 BR BKN003=
ENSB 250950Z 11015KT 9999 FEW025 SCT090 M08/M23 Q1003 NOSIG=
ENTC 250950Z 18017KT 9999 FEW050 BKN080 01/M04 Q0990 NOSIG=
ENTO 250950Z 20014KT 3500 BR VV002 04/03 Q0998=
####018003738####
SAUS70 KWBC 251000
METAR
KPTV 250955Z AUTO 10003KT 6SM OVC047 05/03 A3029 RMK AO1=
KPTW 250954Z AUTO 29005KT 8SM CLR M02/M07 A2998 RMK AO2 SLP153
      T10221072 $=
KPVU 250955Z AUTO 23005KT 10SM OVC040 02/M02 A2986 RMK AO1=
KPWG 250955Z AUTO 35003KT 5SM CLR 04/02 A3043 RMK AO1=
KPWT 250955Z AUTO 21003KT 10SM OVC041 03/01 A3003 RMK AO1=
KRDG 250954Z AUTO A2998 RMK AO2 SLPNO PWINO FZRANO TSNO $=
KRIL 250953Z AUTO 00000KT 10SM BKN090 M04/M08 A2992 RMK AO2 SLP142
      T10441078=
KRKD 250955Z AUTO 32003KT 10SM BKN050 OVC065 M02/M07 A2991 RMK
      AO1=
KRKP 250953Z AUTO 03004KT 4SM BR CLR 09/09 A3036 RMK AO2 SLP281
      T00890089=
KRNH 250955Z AUTO 19003KT 10SM CLR M18/M20 A3037 RMK AO2=
KROG 250955Z AUTO 00000KT 10SM CLR M06/M11 A3044 RMK AO1=

6. The "Collective Message Retrieval" method can be applied to the retrieval of any of the other data types listed in the Openly Accessible Data & Products web page discussion under the File Server Directory Descriptions link.

7. To separate the "bulletins" (containing multiple reports) in the file as shown in the example above, an ASCII message flag field is placed between each WMO bulletin for post processing by the user.



NOTE 1

The WMO "Meteorological Bulletin" is contained within a WMO Meteorological Message. These files on the Gateway servers contain the WMO Meteorological Bulletin portion of the WMO Meteorological Message.

NOTE 2

The suggested use of the UNIX list (ls) command options above [caused by ending the ftp URL with a "/"] would be for the establishment of which observation sites to initially select. All other FTP activity should be direct retrieval of files by file name(s) in the FTP data pull executions of any program or scripts you may write or use. Even the browser FTP activity you generate using this web page will cause additional performance impacts upon the server resources. This is evident by the time delay you experience when you go to the list of filenames in the URLs where it ends with a "/" (see the alert "NOTICE TO FTP USERS" on the Glogal Data and Products web page discussing the use of the UNIX "ls" command)

 

    US Dept of Commerce
    National Oceanic and Atmospheric Administration
    National Weather Service
    Office of Operational Systems (OPS)
    1325 East West Highway
    Silver Spring, MD 20910
    Page Author: NWS Internet Services Team
Disclaimer
Information Quality
Credits
Glossary
Privacy Policy
Freedom of Information Act (FOIA)
About Us
Career Opportunities
     Page last Modified: 4 November, 2013 2:36 PM