Data Processing Descriptions

Which type of data processing would you like to see info about?

Space-Based

Autonomous THEMIS Science Operations Center (SOC) scripts obtain scheduling information from the Mission Operations Center (MOC) and use this information to retrieve and validate space-based instrument raw data files produced during ground station probe contacts. Statistics produced during this process are stored in a MySQL database which is accessed by operations personnel for review. The raw data files are archived on a Redundant Array of Independent Disks (RAID) system for subsequent processing and are backed up onto CD-R media. Retrieval of raw data files triggers production of numerous data products. Initially, the raw data files are converted in Level 0 data files. This includes separating the data by packet Application Identifier (AppID) and archiving these data files on the RAID system, which gives initial low level access by basic data analysis software. The Level 0 files are then converted into Level 1 data files in Common Data Format (CDF). During Level 1 processing, the raw packets are decommutated into various data structures. All known anomalies are corrected during this phase of processing. L1 CDF's have all samples time stamped and the Mission Operations Center (MOC) routinely performs clock correlation tests, and sends clock update commands to keep timing errors below 0.5 sec across the constellation of probes. At this point the data is still raw and un-calibrated, but the CDF format allows wider access and is platform independent. Following creation of Level 1 data files, Level 2 data files are created. This includes calibrated data in physical units and is also in CDF format. Both Level 1 and Level 2 files are archived on the RAID system. Following reception of the raw ground station data files after a contact, the Level 0-2 processing is completed within 1 hour.

Some data types are prone to various sorts of timing errors due to known IDPU and BAU flight software issues. In particular, the requirement to perform accurate coordinate transformations during later Level 2 processing depends on the ability to calculate the (spinning) probe's 3-D spatial orientation at any point in time. A great deal of effort is taken to analyze the sun sensor crossing times (and all known anomalies thereof) from the BAU housekeeping telemetry to produce the Level 1 "state" and "spin model" data products.

To maximize the amount of science data returned from the probes, various packet-level block compression schemes are implemented in the IDPU flight software. All compressed packets are decompressed during the Level 0 processing. It is expected that new compression algorithms will be developed during the mission, as our understanding of the tradeoff between scientific impact and telemetry volume evolves. Any such improvements will be incorporated into the Level 0 processing software.

The Level 0-2 data products are used to produce Summary Data, mainly in the form of data plots which are available via the THEMIS website to the on-duty scientist (Tohban) to assess the quality and state of instrument data collection. Spacecraft ephemeris data are routinely updated and accessed to produce probe state files which are folded into Level 0-2 and summary data processing. These state files are used to produce orbit plots and ground tracks of the probes and are also available via the web site.

The VC (raw) to Level 0 to Level 1 data processing scripts, algorithms and software are under version control in the THEMIS Subversion (SVN) repository under the "tmtools" directory. The processing will be documented in the THEMIS Level 1 File Definition and Developer's Guide Documents.