-
Notifications
You must be signed in to change notification settings - Fork 23
Conversion Tools
Original Author: R. J. Barnes (JHU/APL)
The next generation software uses a self-describing file format designed specifically for SuperDARN. A brief description of the new format is given here. One of the long term aims of SuperDARN has always been to provide data in one of the "standard" formats used by the broader space physics community, such as CDF or netCDF. Incorporated in the new software are the tools to do just that.
One of the great advantages of self-describing formats is that it is possible to write a general purpose converter that translates one format to another - Any changes to the file format do not require the translator to be modified.
The tools included in the SuperDARN software convert from the SuperDARN internal format to netCDF or CDF, two of the most widely used formats. (HDF has a netCDF interface and works quite happily with netCDF files). Converting to these two formats also unlocks a host of other utilities and tools for working with the data.
The SuperDARN format was designed for storage efficiency and so does not include the extra meta-data components that are found in CDF and netCDF, however the software allows the user to add meta-data during the conversion process.
There are some problems with general purpose translators as they do not know of any implicit relationships between items in a data file. For example; SuperDARN data files are composed of records containing the radar operating parameters and observations. The time of each record is stored in the form of one variable each for year,month,day,hour,minute,second and usecond. When converting from the SuperDARN format to CDF, it would be nice to convert this time to a single variable with the type "CDFEpoch", but a general purpose translator is incapable of doing this as it does not understand the relationship between the time variables.
The following is a brief overview and tutorial describing how the new file format data is converted to both netCDF and CDF.
The .fitacf and .rawacf files use the same self-describing file format called dmap. (The illogical reason for this is that the library that works with these files generates a mapping from a data structures into a block of memory).
A useful utility is dmapdump which prints out on the console the contents of a dmap file. This allows you to inspect the contents of a data file:
dmapdump 20021219.kap.fitacf
scalars:
char "radar.revision.major" = 1
char "radar.revision.minor" = 66
short "cp" = 9050
short "stid" = 3
short "time.yr" = 2002
short "time.mo" = 12
short "time.dy" = 19
short "time.hr" = 0
short "time.mt" = 0
short "time.sc" = 0
short "time.us" = 0
short "txpow" = 9000
short "nave" = 67
short "atten" = 0
short "lagfr" = 1200
short "smsep" = 300
short "ercod" = 0
short "stat.agc" = 8192
short "stat.lopwr" = 0
float "noise.search" = 6870
float "noise.mean" = 22870
short "channel" = 0
short "bmnum" = 15
short "scan" = 1
short "offset" = 15
short "rxrise" = 100
short "intt.sc" = 6
short "intt.us" = 0
short "txpl" = 300
short "mpinc" = 2400
short "mppul" = 7
short "mplgs" = 18
short "nrang" = 70
short "frang" = 180
short "rsep" = 45
short "xcf" = 0
short "tfreq" = 14565
int "mxpwr" = 4400000
int "lvmax" = 20000
int "fitacf.revision.major" = 5
int "fitacf.revision.minor" = 0
string "combf" = "$Id: normal_sound.c,v 1.94 2002/10/14 13:00:00 hughes Exp $"
float "noise.sky" = 6707.2
float "noise.lag0" = 0
float "noise.vel" = 0
arrays:
short "ptab" [7]
short "ltab" [2][18]
float "pwr0" [70]
short "slist" [46]
short "nlag" [46]
char "qflg" [46]
char "gflg" [46]
float "p_l" [46]
float "p_l_e" [46]
Note: By default dmapdump only displays the value of scalars but not the values in arrays (arrays are likely to be big), to output the complete file use the -d option: dmapdump -d 20021219.kap.fitacf
The remainder of this tutorial is split into two parts: