1 - UCLA IGPP

Comments

Transcription

1 - UCLA IGPP
thm_soc_123_map
THEMIS Mission Archive Plan
THEMIS
Mission Archive Plan (MAP)
THM-SOC-123_MAP
February 15, 2008
Table of Contents
1.0 Purpose
2.0 Current and Permanent Data Processing and Data Inventory
2.1 Data Processing and Handling
2.1.1 Science Operations Center
2.1.2 Probe Data Processing
2.1.3 Ephemeris Data
2.2 Inventory of Data Products
2.2.1 L2 CDF's Data Products
2.2.2 L2 CDF's Data Descriptions
2.2.3 Summary, ASI Keogram and Tohban Plots
2.3 Availability and Access
2.4 Evidence of Use
2.5 VO's
2.6 Relationship of Metadata to SPASE Data Model
2.7 Resident and Permanent Archives
2.8 Realistic schedule for Enhancements - Summary
3.0 Current and Permanent Inventory, Availability and Access of Spacecraft, Instruments and
Instrument Calibration Documents.
4.0 Current and Permanent Data Analysis Tools
4.1 Trainings
4.2 Web Site
4.3 Science Data Analysis Software
4.3.1 TDAS Description
4.3.2 TDAS Organization
4.3.3. TDAS Version Control
4.4 Graphical User Interface (GUI)
4.5 Permanent Enhancements
4.6 Users Guide
4.7 FTP Site
5.0 Summary
8/3/2017
Page 1 of 12
thm_soc_123_map
THEMIS Mission Archive Plan
1.0
Purpose
The purpose of this document is to provide a plan that defines the current and permanent states of the data products,
data handling, documentation and data analysis tools and defines a reasonable plan to achieve the permanent state of these
products and processes for the end of mission (eom).
2.0
2.1
2.1.1
2.1.1.1
Current and Permanent Data Processing and Data Inventory
Data processing and Handling
Science Operations Center
Overview
The THEMIS Science Operations Center (SOC) retrieves, processes, and archives all data files from the space-based
(probes) and ground-based instrument networks (Ground Based Observatories). All data and data products are freely available
with no passwords or security logons via the THEMIS project website and the THEMIS Data Analysis Software (TDAS). In
addition, numerous external institutions “mirror” some or all of the data and data products.
2.1.1.2 Space-Based Instrument Data Collection And Processing
Autonomous SOC scripts retrieve scheduling information from the Mission Operations Center (MOC) and use this
information to retrieve and validate space-based instrument raw data files produced during ground station probe contacts.
Statistics produced during this process are stored in a MySQL database which is accessed by operations personnel for review.
The raw data files are archived on a Redundant Array of Independent Disks (RAID) system for subsequent processing as well
as being backed up onto CD-R media. Retrieval of raw data files triggers production of numerous data products. Initially, the
raw data files are converted in Level 0 data files. This includes separating the data by packet Application Identifier (AppID)
and archiving these data files on the RAID system. This gives initial low level access by basic data analysis software.
The Level 0 files are then converted into Level 1 data files in Common Data Format (CDF). At this point the data is
still raw and un-calibrated, but the CDF format allows wider access and is platform independent. Following creation of Level 1
data files, Level 2 data files are created. This includes calibrated data in physical units and is also in CDF format. Both Level 1
and Level 2 files are archived on the RAID system. The Level 0-2 data products are used to produce Summary Data, mainly in
the form of data plots which are available via the THEMIS website to the on-duty scientist (Tohban) to assess the quality and
state of instrument data collection. Spacecraft ephemeris data are routinely updated and accessed to produce probe state files
which are folded into Level 0-2 and summary data processing. These state files are used to produce orbit plots and ground
tracks of the probes and are also available via the website. Following reception of the raw ground station data files after a
contact, the Level 0-2 processing is completed within 1 hour.
2.1.1.3 Ground-Based Instrument Data Collection and Processing
Ground based instruments include All Sky Imagers (ASI) and Ground Magnetometers (GMAG). The THEMIS
project, in collaboration with the University Of Calgary, has deployed 20 Ground Based Observatories (GBO) across Canada
and Alaska. Each GBO includes an ASI. Eleven GBOs have a UCLA GMAG. The remaining GBOs take advantage of existing
GMAG networks, using magnetometers co-located near the GBO facility (5 from the University of Alberta CARISMA
network, 2 from the University of Alaska Geophysical Institute GIMA network). The SOC also retrieves the full complement
of GIMA magnetometer data (total of 10 stations) and processes these data into the same products as the UCLA and
CARISMA magnetometer data. Using TDAS software, the University of Alberta will convert the remaining CARISMA
network magnetometer data (total of 13 stations) into the same Level 2 CDF data products and will make those accessible to
external users of the TDAS software.
2.1.1.4 ASI
There are 2 avenues for retrieving the ASI image data. The University of Calgary collects low resolution data daily via
an internet connection to each GBO. An automated RSYNC process at the SOC also runs daily to “mirror” this data on the
RAID system. Once the data are mirrored they are converted into level 2 products in CDF format. Both the low resolution
image files and the L2 data products are available via the THEMIS project website. This data is also accessible via the TDAS
software. The second avenue of collection involves the retrieval of the GBO hard -drives containing the high resolution images.
These hard-drives are collected by the University of Calgary, where the data are downloaded and validated. Once this process
is performed the hard-drives are shipped to the SOC for inclusion in the RAID system.
8/3/2017
Page 2 of 12
thm_soc_123_map
THEMIS Mission Archive Plan
2.1.1.5 GMAG
Automated RSYNC processes run daily to collect the GBO UCLA GMAG data from the University of Calgary, the
GIMA data from the University of Alaska, and the CARISMA data from the University of Alberta, and store these data on the
RAID system. Once the data are retrieved they are converted into level 2 products in CDF format. Daily stack plots are also
made. These data are available via the THEMIS website and the TDAS software
2.1.2
Probe Data Processing
After each probe-ground station contact, any telemetry files acquired by the ground station are transferred to the
THEMIS SOC and automatically dispatched for processing to Level 0 (raw telemetry packets), Level 1 (time-tagged,
uncalibrated data files in CDF format), and Level 2 (calibrated data CDFs in geophysical relevant coordinate systems) files.
All data products are made available to the public via the THEMIS web site immediately after they are produced.
Level 0 packet files are produced by extracting all valid CCSDS source packets from the ground station CCSDS
transfer frame telemetry files. At the completion of this process, the probe L0 archive has been updated with all new packets
found in the telemetry file being processed, and is available to the community. Another automated process runs several times a
day checking for L0 archives which have been updated with fresh telemetry. Any updated L0 data is dispatched for Level 1
processing. To maximize the amount of science data returned from the probes, various packet-level block compression
schemes are implemented in the IDPU flight software. All compressed packets are decompressed during the Level 0
processing. It is expected that new compression algorithms will be developed during the mission, as our understanding of the
tradeoff between scientific impact and telemetry volume evolves. Any such improvements will be incorporated into the Level
0 processing software.
During Level 1 processing, the raw packets are decommutated into various data structures. All known anomalies are
corrected during this phase of processing. The output of the Level 1 processing is a set of CDF files, one file per data type,
each covering approximately one UTC day. The L1 CDFs are available via the THEMIS web site immediately after the L1
processing completes, and at most a few hours after the ground station files are received by the THEMIS SOC. Some data
types are prone to various sorts of timing errors due to known IDPU and BAU flight software issues. In particular, the
requirement to perform accurate coordinate transformations during later Level 2 processing depends on being able to calculate
the (spinning) probe's 3-D spatial orientation at any point in time.
A great deal of effort has been put into analyzing the sun sensor crossing times (and all known anomalies thereof) from the
BAU housekeeping telemetry to produce the Level 1 "state" and "spin model" data products.
2.1.3
Ephemeris Data
During normal operations THEMIS orbit solutions are generated on Mondays, Wednesdays, and Fridays each week.
30-day predictive ephemerides, including upcoming maneuver operations, are generated from these orbit solutions for
scheduling and operational use. Definitive ephemerides are created from the orbit solution archive on a weekly basis, covering
the previous calendar week.
2.2
2.2.1
Inventory of Data Products
L2 CDF's Data Products
Level 2 THEMIS CDF files contain THEMIS calibrated data quantities in physical units. These data can be used by
scientists directly; the instrumental details have been accounted for in the calibration process. Level 2 data files are stored in
the permanent archive at UCB and are distributed to mirror sites and the SPDF. Level 2 files are updated and reprocessed
when necessary. For example, updates in calibration data for a given instrument/time period will result in new Level 2 data
files for that instrument and time period. Level 2 data files are created daily using Level 1 data and calibration data. Level 2
files are also updated and reprocessed when necessary. For example, updates in calibration data for a given instrument/time
period will result in new Level 2 data files for that instrument and time period. Currently (as of 1-Jan-2008), there are Level 2
files for ESA, SST, FBK, and FGM data. In the near future (early 2008), daily processing of EFI, FIT, MOM and SCM Level 2
data files will commence. The full set of Level 2 data files will be available by July 2008. A full list of the available Level 2
data quantities is shown in the following section (L2 CDF Data Descriptions). Please note that the data variables currently
available have an '*' in the 'Curr' column and those to be provided before end of mission (permanent) have an '*' in the 'Perm'
column.
8/3/2017
Page 3 of 12
thm_soc_123_map
THEMIS Mission Archive Plan
2.2.2
L2 CDF's Data Descriptions
Instrument
ESA
SST
MOM
(on-board
moments)
8/3/2017
Data Name
For ESA: ? = f or r or b
pei?_density
pei?_t3
pei?_p3
pei?_en_eflux
pei?_velocity_dsl
pei?_velocity_gse
pei?_velocity_gsm
pei?_ptens
peif_mftens
pei?_magt3
pei?_avgtemp
pei?_vthermal
pei?_symm
pei?_symm_ang
pei?_tot
pei?_en_counts
pei?_mode
pee?_density
pee?_t3
pee?_t3
pee?_en_eflux
pee?_velocity_dsl
pee?_velocity_gse
pee?_velocity_gsm
pee?_ptens
pee?_mftens
pee?_magt3
pee?_avgtemp
pee?_vthermal
pee?_symm
pee?_symm_ang
pee?_ang
pee?_tot
pee?_en_counts
pee?_mode
For SST: ? = f or r or b
psi?_density
psi?_t3
psi?_p3
psi?_en_eflux
psi?_velocity_dsl
psi?_velocity_gse
psi?_velocity_gsm
psi?_ptens
psif_mftens
psi?_magt3
psi?_avgtemp
psi?_vthermal
psi?_symm
psi?_symm_ang
psi?_tot
psi?_en_counts
psi?_mode
pse?_density
pse?_t3
pse?_t3
pse?_en_eflux
pse?_velocity_dsl
pse?_velocity_gse
pse?_velocity_gsm
pse?_ptens
pse?_mftens
pse?_magt3
pse?_avgtemp
pse?_vthermal
pse?_symm
pse?_symm_ang
pse?_ang
pse?_tot
pse?_en_counts
pse?_mode
peim_density
peim_flux
peim_mftens
peim_eflux
peim_velocity
peim_press
peem_density
peem_flux
peem_mftens
peem_eflux
peem_velocity
peem_press
psim_#
psem_#
Description
Curr
f=full, r=reduced, b=burst
ion density
diagonalized ion temperature
diagonalized ion pressure
ion energy flux spectrogram
ion velocity (DSL)
ion velocity (GSE)
ion velocity (GSM)
ion pressure tensor (DSL)
ion momentum flux tensor (DSL)
ion temperatures, parallel and perp to B field
trace of diagonalized temperature tensor divided by
ion thermal velocity
direction of pressure tensor symmetry (DSL)
angle between symmetry direction and B
total ion count
ion count vs. energy
ion instrument mode
electron density
diagonalized electron temperature
diagonalized electron pressure
electron energy flux spectrogram
electron velocity (DSL)
electron velocity (GSE)
electron velocity (GSM)
electron pressure tensor (DSL)
electron momentum flux tensor (DSL)
electron temperatures, parallel and perp to B field
trace of diagonalized temperature tensor divided by
electron thermal velocity
direction of pressure tensor symmetry (DSL)
angle between symmetry direction and B
electron angle spectrogram
total electron count
electron count vs. energy
electron instrument mode
f=full, r=reduced, b=burst
ion density
diagonalized ion temperature
diagonalized ion pressure
ion energy flux spectrogram
ion velocity (DSL)
ion velocity (GSE)
ion velocity (GSM)
ion pressure tensor (DSL)
ion momentum flux tensor (DSL)
ion temperatures, parallel and perp to B field
trace of diagonalized temperature tensor divided by
ion thermal velocity
direction of pressure tensor symmetry (DSL)
angle between symmetry direction and B
total ion count
ion count vs. energy
ion instrument mode
electron density
diagonalized electron temperature
diagonalized electron pressure
electron energy flux spectrogram
electron velocity (DSL)
electron velocity (GSE)
electron velocity (GSM)
electron pressure tensor (DSL)
electron momentum flux tensor (DSL)
electron temperatures, parallel and perp to B field
trace of diagonalized temperature tensor divided by
electron thermal velocity
direction of pressure tensor symmetry (DSL)
angle between symmetry direction and B
electron angle spectrogram
total electron count
electron count vs. energy
electron instrument mode
ESA ion density
ESA ion flux
ESA ion momentum flux tensor
ESA ion energy flux
ESA ion velocity
ESA ion pressure
ESA electron density
ESA electron flux
ESA electron momentum flux tensor
ESA electron energy flux
ESA electron velocity
ESA electron pressure
# = SST quantities (same as for ESA)
# = SST quantities (same as for ESA)
Perm
*
*
*
3
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
3
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
3
*
3
Page 4 of 12
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
thm_soc_123_map
THEMIS Mission Archive Plan
Instrument
EFI
FBK
FFT
(on-board)
FGM
FIT
(on-board)
GMAG
SCM
STATE
8/3/2017
Data Name
eff
efp
efw
eff_dot0
efp_dot0
efw_dot0
eff_0
efp_0
efw_0
efs
efs_0
efs_dot0
vaf
vap
vaw
vbf
vbp
vbw
fb1
fb2
fbh
fb_hff
fb_eac12
fb_eac34
fb_eac56
fb_edc12
fb_edc34
fb_edc56
fb_scm?
fb_v?
ffp_16
ffp_16_dbpara
ffp_16_dbperp
ffp_16_eac12
ffp_16_eac34
ffp_16_eac56
ffp_16_edc12
ffp_16_edc34
ffp_16_edc56
ffp_16_epara
ffp_16_eperp
ffp_16_scm?
ffp_16_v?
ffp_32_#
ffp_64_#
ffw_16_#
ffw_32_#
ffw_64_#
fgl_dsl
fgl_gse
fgl_gsm
fgh_dsl
fgh_gse
fgh_gsm
fgs_dsl
fgs_gse
fgs_gsm
efs
efs_0
efs_dot0
efs_sigma
fgs
fgs_sigma
fit
fit_bfit
fit_efit
mag_????
For SCM: ? = f or p or w
scf
scp
scw
state_pos
state_pos_gse
state_pos_gsm
state_vel
state_vel_gse
state_vel_gsm
state_att
state_att_gse
state_att_gsm
state_man
state_roi
state_spinras
state_spindec
state_spinalpha
state_spinbeta
state_spinper
state_spinphase
Description
exists
E field, fast survey/full orbit, 3D
E field, particle burst, 3D
E field, wave burst, 3D
E field, fast survey/full orbit, 3D, using E dot B=0
E field, particle burst, 3D, using E dot B=0
E field, particle burst, 3D, using E dot B=0
E field, fast survey/full orbit, 3D, using Ez=0
E field, particle burst, 3D, using Ez=0
E field, particle burst, 3D, using Ez=0
On-board spin-fit electric field
On-board spin-fit electric field using Ez=0
On-board spin-fit electric field using E dot B=0
Voltage, processor A, fast survey/full orbit
Voltage, processor A, particle burst
Voltage, processor A, wave burst
Voltage, processor B, fast survey/full orbit
Voltage, processor B, particle burst
Voltage, processor B, wave burst
Filter Bank 1 (E and/or B)
Filter Bank 2 (E and/or B)
Filter Bank high frequency (100-300kHz)
High-frequency filter peak and average values
Spectrogram E field AC component, sensors 1&2 (spin plane)
Spectrogram E field AC component, sensors 3&4 (spin plane)
Spectrogram E field AC component, sensors 5&6 (axial)
Spectrogram E field DC component, sensors 1&2 (spin plane)
Spectrogram E field DC component, sensors 3&4 (spin plane)
Spectrogram E field DC component, sensors 5&6 (axial)
Spectrogram SCM? (search coil) ; ?=1,2,3 (three axes)
Spectrogram floating potential of sensor ?=1,2,3,4,5,6
FFT power spectrum in particle burst x 16 frequencies
FFT power spectrum for dB (parallel)
FFT power spectrum for dB (perpendicular)
FFT power spectrum for AC component E12
FFT power spectrum for AC component E34
FFT power spectrum for AC component E56
FFT power spectrum for DC component E12
FFT power spectrum for DC component E34
FFT power spectrum for DC component E56
FFT power spectrum for E (parallel)
FFT power spectrum for E (perpendicular)
FFT power spectrum for SCM? ; ?=1,2,3 (axes)
FFT power spectrum for V? ; ?=1,2,3,4,5,6 (sensors)
# = same quantities in particle burst x 32 frequencies
# = same quantities in particle burst x 64 frequencies
# = same quantities in wave burst x 16 frequencies
# = same quantities in wave burst x 32 frequencies
# = same quantities in wave burst x 64 frequencies
B field, low telemetry (DSL)
B field, low telemetry (GSE)
B field, low telemetry (GSM)
B field, high telemetry (DSL)
B field, high telemetry (GSE)
B field, high telemetry (GSM)
B field, spin-resolution magnetic field B (DSL)
B field, spin-resolution magnetic field B (GSE)
B field, spin-resolution magnetic field B (GSM)
On-board spin-fit electric field (EFI) data
On-board spin-fit electric field (EFI) using Ez=0
On-board spin-fit electric field (EFI) using E dot B=0
Variance of onboard spin-plane electric field spin fit
On-board spin-fit FGM data
Variance of onboard spin-plane magnetic field spin fit
SpinFIT file E&B raw data
FGM spinfit calibrated data: A,B,C,sig,avg
EFI spinfit calibrated data: A,B,C,sig,avg
Ground magnetometer data in DHZ coordinates
(???? = 4-letter code of ground station)
f=fast survey, p=particle burst, w=wave burst
waveform fast survey (DSL)
waveform particle burst (DSL)
waveform wave burst (DSL)
GEI position, xyz
GSE position
GSM position
GEI velocity, xyz
GSE velocity
GSM velocity
GEI attitude
GSE attitude
GSM attitude
Maneuver flag
Regions of interest
spin axis right ascension, deg
spin axis declination, deg
Geom to spin axis, Euler alpha, deg
Geom to spin axis, Euler beta, deg
spin period, sec
spin phase, deg
Page 5 of 12
desired
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
thm_soc_123_map
THEMIS Mission Archive Plan
2.2.3
Summary, ASI Keogram and Tohban Plots
Summary plots of the THEMIS data quantities are provided online, and are available from the THEMIS web site at
http://themis.ssl.berkeley.edu/summary.php. A number of different plots are available. For each probe for each day there are a
total of five single-probe overview plots: one covering the full day, and four others covering 6 hour time periods. Each singleprobe overview plot contains ground based magnetometer data, all-sky imager keogram data, FGM magnetic field data, ESA
density and velocity measurements, and energy flux spectrograms for ions and electrons, SST energy flux spectrograms fr ions
and electrons, and dynamic power spectra from FBK. Also there is a status bar which shows whether the spacecraft is in slow
survey mode, fast survey mode or burst mode.
For the ESA, FGM and SST instruments, we provide summary plots for all of the probes for a given day. The FGM
plots show fgl (low telemetry rate) and fgs (spin fit) magnetic field data in GSE coordinates, and also a status bar for each
spacecraft. For ESA the electron and ion energy flux spectrograms are shown for each probe, for full, reduced and burst modes.
For SST the electron and ion energy flux spectrograms are shown for each probe, for full and reduced modes. For each plot
type (FGM, ESA Full, ESA Reduced, ESA Burst, SST Full, and SST Reduced) for each day there are five plots, one covering
the full day, and four which cover 6 hour periods.
THEMIS also provides summary plots for ground based magnetometer data. These include stacked plots of the 3
components of the magnetic field for the THEMIS GBO magnetometer stations for 16 high latitude stations and 6 low latitude
stations. For all-sky imager data there are three different types of overview plot. The all-sky image Summary plot is an image
with hours from left to right, and 20 THEMIS GBO stations running from top to bottom with the westernmost station on top
and the easternmost station at the bottom. A small image is shown if there is data for a particular hour and station. The
Keogram plots show daily keograms that present the aurora with slightly more detail. The Average plot shows a series of
images that were created by averaging full-resolution images for a minute at the GBO site and transmitted, one per hour. These
images are intended to allow assessment of the camera and observing conditions. The spacecraft position is shown in the
summary GSM Orbit plots. These show the X, Y, and Z positions of each probe in GSM coordinates. The footprints over the
north and south polar regions are shown in the Ground Tracks North and South plots. These plots also show the GSE
coordinate positions. An example of a Summary Plot is below.
8/3/2017
Page 6 of 12
thm_soc_123_map
THEMIS Mission Archive Plan
2.3
Availability and Access
All data and data products are freely available via the THEMIS project website (see section 4.1) and the THEMIS
Data Analysis Software (TDAS - see sections 4.2 and 4.3). In addition, numerous external institutions “mirror” some or all of
the data and data products. The Space Physics Data Facility (SPDF) mirrors the Level 2 data created at the THEMIS SOC and
makes that available to the wider space physics community via their CDAWeb and SSCWeb websites. Based on user input to
various IDL routines included in the THEMIS Data Analysis Software (TDAS), a HTTP connection is made to the THEMIS
SOC RAID and Level 1 or Level 2 CDF’s are downloaded for subsequent viewing and processing. Similarly, the National
Central University’s Institute of Space Science in Taiwan maintains a complete ascii and IDL data archive of the Level 2 data.
The Centre de Donnees de la Physique des Plasmas in France provides a complete set of L0, 1, and 2 data sets in CDF format.
Consequently, the full THEMIS data sets are backed-up.
2.4
Evidence of Use
The THEMIS Science Data Product's and Web Page's usage at the Space Science Laboratory at the University of
California at Berkeley is denoted in the graphs below. In addition through December 4,2007 there were almost 14,000 requests
from SPDF's CDAWeb representing the transfer of 40Gb of THEMIS data and at the SSCWeb over 3,000 requests to see
THEMIS Orbit information.
2.5
VO's
The goal of the VMO is to facilitate query-based discovery and access of past, present, and future NASA high-altitude
Magnetospheric missions. The core function of the VMO data environment is to search and retrieve pointers to Magnetospheric
data while presenting the user with a common interface, either a web interface or an application programming interface (API).
The THEMIS mission, launched in February 2007, is part of this effort.
Dr. Vassilis Angelopoulos, a VMO Co-Investigator, will supervise the scientific aspects of the effort, in terms of data
quality and compatibility of the submitted datasets with the other datasets in the VMO database for maximum science return.
His programming staff will describe the THEMIS data sets in SPASE+ (Space Physics Archive Search and Extract) terms and
set up a data service for VMO. They will co-develop and test the SOAP (Simple Open Access Protocol) interface for their data
service. They will facilitate interactions between the distributed THEMIS team (UCB, LASP, CETP, TUBS) and the VMO
group. Dr. Angelopoulos will be a member of the VMO SPASE+ definition group. The VMO effort was started on 11/2007
and is scheduled to be completed by 9/2008. A second proposal for 10/2008 – 9/2009 may follow depending on progress
during the first year.
2.6
Relationship of Metadata to SPASE Data Model
The Themis meta data is currently stored in the cdf's that serve as the primary means of Themis data distribution.
Most of the attributes required by the SPASE model can be found in these cdf's. Any additional meta data that is required can
be found in text based mission documents. The cdf data does not, however, have the hierarchal structure of SPASE xml.
SPASE meta data is generated by, first, using the online CDF to SPASE xml converter. This will turn the flat cdf structure into
a hierarchal SPASE xml structure. Then the meta data is inspected by hand and any missing data is added.
Finally, the meta data is validated using the online validation tools and also inspected for any errors.
VxO's SPASE xml files are going to be made available to the Goddard VMO and the UCLA VMO as needed.
Later middleware to search Themis data products and possibly dynamically generate higher level data products will be made
available as needed. The requirements of each VxO will be identified and met by collaboration between Goddard Space Flight
Center (GSFC), University of Maryland, University of California at Berkeley and Los Angles.
8/3/2017
Page 7 of 12
thm_soc_123_map
THEMIS Mission Archive Plan
Currently draft SPASE xml files are currently available for Themis-A Fluxgate Magnetometer (FGM), Themis-A Probe Status,
FGM Measurements 1/128s and Themis A Probe State. The GSFC VMO is currently is in the process of making this data
available. Permanent (future) work will expand the data available to all probes and all instruments.
2.7
Resident and Permanent Archives
All THEMIS data and data products are archived on the RAID system. Currently, this includes approximately 5.5
Terabytes of data. Autonomous scripts archive raw space-based data to CD-R media as soon as 650 Mbytes accumulate
(~every 5 days). The full complement of GBO data (ASI and UCLA GMAG) are retrieved by the University of Calgary and
archived on a RAID system. UCLA and the University of Alberta run daily RSYNC jobs to retrieve and archive all GMAG
data resident on the THEMIS SOC RAID. The CETP in France runs a daily RSYNC job to retrieve and archive all THEMIS
data and products on the THEMIS SOC RAID system.
The archive of all on-orbit L0 packet data is publicly available via the THEMIS website:
http://themis.ssl.berkeley.edu/data/themis/PROBE/l0/YYYY/MM/DD/ where PROBE is one of tha,thb,thc,thd,the, or thf, and
YYYY/MM/DD denotes the UTC date. Each directory comprises a set of raw packet files, decompressed if necessary, one file
per apid, with header timestamps occurring on the specified date. The L1 on-orbit probe data is accessed via the same site:
http://themis.ssl.berkeley.edu/data/themis/PROBE/l1/DATATYPE/YYYY. PROBE is defined the same as above, DATATYPE
corresponds to one of the available THEMIS L1 data types (see section x.xx.), and YYYY is the year. The directory contains
all the CDFs of that datatype for the specified year.
2.8
Realistic schedule for Enhancements - Summary
There are no plans for any significant changes to the THEMIS SOC, L0 and L1 Data Processing. There are new data
variables that will be added to the existing ESA, SST, FBK, FGM and State L2 cdf's. There will new L2 cdf's created for MOM
(on-board), EFI, FFT (on-board), FIT (on-board), GMAG and SCM. All the revisions to the existing L2 cdf's and the
development of the new L2 cdf's will be completed by 12/2008. THEMIS data sets will be described in SPASE+ terms and a
VMO data service set up a data service and develop and test the SOAP (Simple Open Access Protocol) interface for the
THEMIS data service all by 9/2008. Resident and Permanent Archives are setup and no significant change are anticipated. All
Documentation currently not completed is scheduled to be completed during 2008. Significant revisions to the THEMIS Data
Analysis Tools as descibed in section 4.5 (Permanent Enhancements) will also be completed by 9/2009.
3.0
3.1
Current and Permanent Documentation Inventory
Inventory, Availability and Access of Spacecraft, Instruments and Instrument Calibration Documents
The current and permanent inventory of THEMIS documentation is listed below. Access to all THEMIS
documentation is fully open with no accounts, passwords or registration procedures required. The document resides on an ftp
site that can be accessed from the THEMIS Documentation web page http://themis.ssl.berkeley.edu/software_docs.shtml.
8/3/2017
Page 8 of 12
thm_soc_123_map
THEMIS Mission Archive Plan
Document Name
thm_sys_012_pdmp
thm_soc_101_time
thm_soc_102_statefile
thm_soc_103_hsk_varnames
thm_soc_105_fields_L1_varnames
thm_soc_106_particles_L1_varnames
thm_soc_108_gmag_L1_varnames
thm_soc_109_asi_L1_varnames
thm_soc_110_coordinates
thm_soc_111_sunsensproc
thm_soc_112_attpaiproc
thm_soc_113_fgm_calproc
thm_soc_114_scm_calproc
thm_soc_115_efi_calproc
thm_soc_116_esa_calproc
thm_soc_117_sst_calproc
thm_soc_119_asi_calproc
thm_soc_120_ analysis_sfw_users_guide
thm_soc_121_ analysis_sfw_developers_guide
thm_soc_122_ administrators_guide
thm_soc_123_ map
thm_soc_124_ L1_file_def
thm_soc_125_ L2_file_def
thm_soc_126_ data_var_name_desc
8/3/2017
Document Description
THEMIS Project Data Management Plan
THEMIS Time Definition
THEMIS Probe State File Definition
THEMIS Housekeeping L1 Data Files Variable
THEMIS Fields Variable Names
THEMIS Particles Variable Names
THEMIS GMAG Variable Name Def’s
THEMIS ASI Variable Name Definitions
THEMIS Coordinate Systems Definitions
THEMIS Sun Sensor Science Processing
THEMIS Science ATT & Inertia Determ.
THEMIS FGM Calibration File and Processing
THEMIS SCM Calibration File and Processing
THEMIS EFI Calibration File and Processing
THEMIS ESA Calibration File and Processing
THEMIS SST Calibration File and Processing
THEMIS ASI Calibration File and Processing
THEMIS Data Analysis Software User's Guide
THEMIS Software Developer's Guide
THEMIS Software Administrators Guide
THEMIS Mission Archive Plan
THEMIS L1 File Definitions
THEMIS L2 File Definitions
THEMIS Data Variable Names Descriptions
Status
Draft on THEMIS ftp site. Final due 12/2008
Completed and on THEMIS ftp site
Completed and on THEMIS ftp site
Completed and on THEMIS ftp site
Completed and on THEMIS ftp site
Completed and on THEMIS ftp site
Completed and on THEMIS ftp site
Completed and on THEMIS ftp site
Completed and on THEMIS ftp site
Completed and on THEMIS ftp site
Completed and on THEMIS ftp site
Completed and on THEMIS ftp site
due 04/2008
due 12/2008
due 12/2008
due 12/2008
Completed and on THEMIS ftp site
Completed and on THEMIS ftp site
due 04/2008
due 04/2008
In progress - due 02/2008
In progress - due 01/2008
due 02/2008
Draft on THEMIS ftp site. Final due 2/2008
Page 9 of 12
thm_soc_123_map
THEMIS Mission Archive Plan
4.0
4.1
Current and Permanent Data Analysis Tools
Trainings
Trainings on the Data and Software availability via the THEMIS Web Site were held at UC Berkeley ,
UCLA (2), University of Colorado, GEM 2008 in San Francisco and the THEMIS SWG. The THEMIS Web Site, installation
of the Software, downloaded data, use of the Graphical User Interface (GUI) and availability of information in the THEMIS
User's Guide were reviewed. In total over 200 people attended these Trainings.
4.2
Web Site
A wealth of information is available via the THEMIS website at http://themis.ssl.berkeley.edu/index.shtml. From the
'Mission' drop down menu information on the Spacecraft, Instruments, Orbits, Status as well as link to many of our domestic
and foreign partners. Under the 'Data' drop down menu access is available to list and description of all THEMIS data variables
and their descriptions. In addition access to a summary plot viewer is available to look at individual probe, multi-probe,
ground, and orbit data. Summary plots are available within 1 hour of reception of data. Also, Level 1 and Level 2 CDF files are
available for download via a data download web page. From the 'Software' drop down menu access is available to the TDAS
Software, Software Users Guide, all THEMIS Documentation and a the list of Monthly THEMIS Enhancements. The
'Publication' drop down menu will link to the THEMIS Publication ftp sit. All of the THEMIS Data, Software and FTP sites are
totally open and available without any logon's or passwords required.
4.3
4.3.1
Science THEMIS Data Analysis Software (TDAS)
TDAS Description
The THEMIS Data Analysis Software (hereafter referred to as TDAS) is an IDL-based software package that can be
used to download, open, analyze and plot Level 1 or Level 2 data and process Level 1 data into Level 2 data quantities. The
TDAS software is accessed via the IDL command line interface and a Graphical User Interface (GUI) developed at SSL. For
users that do not have an IDL license, the THEMIS software development team will provide a version of the GUI package that
can be called from the IDL virtual machine, which does not require a license.
4.3.2
TDAS organization
The TDAS software tree is divided into three branches. The "themis" branch contains software that is used for
THEMIS data extraction and analysis. The "ssl_general" branch contains general use and utility software that is designed for
data analysis of space science data. This software has been used extensively for previous missions such as FAST, WIND and
POLAR. The software in the first two branches has been developed and is currently maintained at SSL. The third branch,
"external", is reserved for software that is developed elsewhere and distributed as part of the TDAS package.
Currently this branch contains the CDAWeb software package developed by SPDF for display and analysis of data contained
in CDF-format files, and the IDL-GEOPACK package, which is an IDL application for N.A. Tsyganenko's magnetic field
modeling package.
8/3/2017
Page 10 of 12
thm_soc_123_map
THEMIS Mission Archive Plan
4.3.3
TDAS Version Control
TDAS version control and software releases Software management and version control is accomplished using the
Subversion version control system (http://subversion.tigris.org/). All versions of the software are contained in a passwordprotected repository. Software developers and scientists submit code and code revisions to this repository and the software is
exported from the repository in a series of "releases". A major release (e.g., TDAS 2.0, 3.0) is issued approximately once every
three months, following an extensive QA process. Occasionally, bugs are found in the released version; when this occurs the
bugs are fixed and a minor release is issued. Also, upgrades and enhancements that are deemed to be useful to the community
(such as improvements in calibration software) may be included in a minor release. The current version of the released version
of the software is v3.02. In addition to the releases, a snapshot of the subversion repository is created every night and is made
available on the THEMIS web site, which is described below. This "bleeding edge" software package always contains the most
up-to-date version of the TDAS package, but is only unit tested. A new "bleeding edge" version is built and released every
night. The user community has a choice of using the latest released version or the latest "bleeding edge" version. Both can be
freely downloaded from the THEMIS Software web page.
4.4
TDAS Command Line and GUI
The IDL command line environment gives the user the full capability of the TDAS package, but a familiarity with
IDL and the IDL-based "tplot" package used for data analysis is necessary. The command line interface is extensively
documented in the THEMIS User's Guide: ftp://apollo.ssl.berkeley.edu/pub/THEMIS/ 3 Ground Systems/3.2 Science
Operations/ Science Operations Documents/Software Users Guides/ and there are many examples in the form of crib sheets
distributed with the software. For users who are not familiar with IDL, the THEMIS software development team has created a
Graphical User Interface (GUI). The THEMIS GUI is built on top of the command line software. It provides full capability for
loading and plotting THEMIS data and also support for some data analysis tasks, such as coordinate transformations, timeseries analysis and power spectra. The THEMIS Overview plots which are displayed online as also included as a GUI option,
for cases where the user wants to specify the time range of the plot. The main THEMIS GUI widget is displayed below.
4.5 Permanent Enhancements
The enhancements listed in this section are scheduled to be completed by 9/2009 (eom). The THEMIS Web Site will
be enhanced to include an Automatic Paper Publication capability as well listing of the following types of events: Substorms as
identified by GBO data, Important Events, Flux Transfer (FTE), Reconnection. The TDAS Software will be enhanced to
incorporate data from the following other missions: GOES, LANO and CLUSTER. In addition the following ancillary Ground
Base Observations will be included: MAGDAS Network, Aura Electrolets (AE), Ground Raiders, SuperDarn, AMISR and
8/3/2017
Page 11 of 12
thm_soc_123_map
THEMIS Mission Archive Plan
SONDE. The Graphical User Interface (GUI) will be enhanced with the following significant features: Buttons for PostScript,
Tohban Plots and interface with SPLASH and new Coordinate Transformation Options. Warning messages will be added to
improve it's easy to use standard.
4.6
THEMIS Science Software User's Guide
The THEMIS Science Software User's Guide presents to the worldwide scientific community what tools are available
for viewing, downloading, processing, calibrating and plotting THEMIS data and how to use these tools. This document is
available on the THEMIS ftp site at UC Berkeley and can be viewed or downloaded from the THEMIS Web site Software web
page.
4.7
THEMIS FTP Site
The THEMIS ftp site is a freely accessed, no logon or password site. It contains the THEMIS Documents,
Publications, Science Data Analysis Tools Enhancement Reports, Training Presentations and Meetings information. The
THEMIS ftp site (ftp://apollo.ssl.berkeley.edu/pub/THEMIS/) is a directory tree served by SPRG's main file server, apollo.
Our backup server performs incremental saves of the THEMIS ftp site nightly using the EMC Networker backup software.
A full backup of the THEMIS ftp site is performed on the 4th day of each month. Backups of the THEMIS ftp site are
retained on tape for four months. One or more these backups may be retained permanently upon request.
5.0
Summary
The THEMIS Mission Archive Plan (MAP) has noted the current state of the THEMIS data products, data handling,
documentation and data analysis tools. For all of the these areas the MAP has clearly described the current state as well as what
is left to do between now and the possible end of mission in 9/2009. In addition the MAP sets out as directed, a reasonable plan
to complete all the permanent not yet accomplished data products, data handling, documentation and data analysis tools.
8/3/2017
Page 12 of 12

Similar documents