General XDC IDPC
General External Data Center Instrument Data Processing
Circuit
12/12/96
Currently, many processes produce some form of quality checks on the data as it is passed through the system. These checks are sent to the user "operx" in the form of e-mail, and some checks are sent to logs. We do not have a central log like the site operations log located at SGP or a central database. However, the e-mail collections and log files currently act a central source of QA information.
This document is based on IDPC documentation found on the SDS Documentation page. For an example of a specific XDC IDPC see the Oklahoma Mesonet IDPC.
Description: Collections usually are done by c-shell scripts which fetch data from the external sites. These c-shell scripts usually ftp the data files from the external site, rename them to the appropriate ARM name and transfer them to the appropriate data directory. There are also some quality checks done at this point. The scripts may report missing files, wrong time stamps, or that files are of an unexpected size. The scripts may do some "filtering" on the data. This filtering may include merging data or extracting subsets.
Description:
Filters on XDC data are processes run on raw data to transform the
raw data to a format that is either more useful to XDC ingesting
processes or is more easily handled throughout the ARM project. For
example, some data is broken down from a merged monthly file to
smaller files each containing 24 hours worth of data. With other
data, only a subset of the raw data collected is of interest to the
ARM community and therefore that subset (indicated by geographical
coordinates) will be extracted from the raw data.
These filters are sometimes combined with collections. Other times, there may be separate processes used to reformat the data.
Description:
The files check is done nightly, and checks for expected number
and sizes of incoming raw data files and ingested files. The
results from this check is sent to operx via e-mail and is updated
on the Web at our Daily XDC file check page at
http://www.xdc.arm.gov/data/prod/public_reports/filecheck/armxdc_check.html.
Description: Here our incoming data files are transformed into either netCDF or HDF format. The ingests may do some checks as far as quality is concerned. Checks are done for missing data or out of range data. Most of the messages generated are sent to a logfile, either a general logfile created by the Zebra Eventlogger or to a log file specified within the ingest. The Zebra logfile is also sent to operx on a daily basis via e-mail and should soon be put up on the our daily status reporting web page.
Description : QME is quality assurance done by ingests and scripts to access the quality of the processed data. So far this is one area of the general XDC IDPC which probably is need of the most attention. Some QA is done within ingests and VAPs, where min and max data points are recorded in the NetCDF headers. Other QA plans include created daily and interactive plots of the NetCDF data produced by the XDC. Currently these checks are sent to operx via e-mail, kept in logfiles, occassionally listed within the NetCDF file itself, and listed in various XDC Web Pages.
Description :
Packing and shipping at the XDC basically is the process of
specifying the appropriate files to be sent on a daily basis to the
Archive and the Experiment Center. This also includes the gathering
and shipping of satellite data shipped to NASA Langley in care of
Bill Smith.