Experimentreadr and SpectraQuant Filtering: Difference between pages

From Eigenvector Research Documentation Wiki
(Difference between pages)
Jump to navigation Jump to search
imported>Jeremy
 
imported>Jeremy
No edit summary
 
Line 1: Line 1:
===Purpose===
CPSA is a preprocessing method used to remove interferences from data prior to building a regression model. The CPSA filter includes one or more of the following corrections:


Read an Experiment File containing filenames and reference measurement data, and imports the corresponding files and data into Analysis GUI. Makes it possible to automatically assign the data to x and y and to calibration and validation blocks.
# multivariate filtering of components defined by a set of clutter or "correction" spectra.
# detrending (aka baselining) of the input data in windows of variables.
# an integrated thickness/pathlength correction.


===Synopsis===
===CPSA Panel===


:experimentreadr(''filename'')
The CPSA tools are accessed through the CPSA panel of Analysis (click on the "CPSA Settings" button just below the top status panel). The CPSA settings panel is only available when using the PCR or PLS Analysis methods and only enabled when calibration data has already been loaded. The default settings for the panel are shown below.


===Description===
[[Image:Cpsa_panel_empty.png]]


Experiment files include a list of data files and their corresponding
"properties of interest" (y-values). An experiment file is expected to be
a plain text file (comma, space, tab or other delimited file) or a Microsoft
Excel-formatted file. The file must consist of one column of text strings indicating the files to be
read and used as samples in the X-block of a regression or classification
model and at least one column of numerical values indicating the values to use as
the corresponding y-block. If no experiment file filename is specified on
the command line, the user is prompted to locate a suitable file.


Once loaded, the experiment file can be manipulated, excluding samples
Four different options categories are available on the panel:
using the include field of the Row Labels tab, or y-block columns using
the include field of the Column Labels tab. Samples can be marked as in
the Calibration or Validation set using the Row Labels tab.


When all manipulations are complete, the user clicks the check-mark
* '''Correction Spectra''' - Allows selecting correction spectra which describe background components which should be removed from the spectra prior to analysis.
toolbar button to import all the indicated files and automatically load
the experiment data into the Analysis GUI.  


====X-block File Formats====
* '''Regions/Baselines''' - Defines regions of the spectra and the baseline order to apply to each region.
The x-block files named can be in any standard readable
file format. However, experiment files do not currently allow for any
multi-file formats. Named files must contain only one sample (row) of
data per file.


====Header Row====
* '''Thickness Correction''' - Determines whether or not to use thickness/pathlength correction (requires thickness/pathlength reference in calibration data)
An experiment file can include an optional header row for the
filenames and properties of interest. This row can contain text lables
which will be used to label the y-block columns (i.e. giving a text
description of the property of interest.)


====Calibration/Validation====
* '''Force PC80 Compatible Preprocessing''' - Determines whether to enforce strict preprocessing rules on the created model.
Experiment files can also contain information
used to split the data into calibration and validation sets. To use this
feature, include an additional column with the keywords "Calibration" or
"Validation" next to each file. When the experiment is imported, the data
will be automatically loaded into the appropriate data blocks.
NOTE: other valid synonyms include (all are case insensitive)
:Calibration = Cal = C   
:Validation  = Val = V = Test = T


====Overriding File Format====
If the extension on the specified files does not unambiguously identify the importer to be used (e.g. xy files with an
extension of ".txt" will be read by the delimited text file importer, not the XY importer), then the file may supply an additional "header" line above the column headers which specifies the file format to expect. This line must contain the keyword "format" followed by an equal sign and the name of the import method to use. For example:
  format=xy
Note that an overriding file format can ONLY be specified when a column header row (described below) is also included.


===Example Experiment File===
===Correction Spectra===
<tt>
  filename,concentration,cal/val
  file1.spc,13.2,cal
  file2.spc,19.0,cal
  file3.spc,5.3,cal
  file4.spc,8.3,val
</tt>


The above experiment file would define an experiment with three samples with X-block data
The list on the left of this option shows the file(s) which have been selected by the user as correction spectra (aka background or clutter spectra) which should be removed from all calibration and test spectra. To perform the filter, the correction spectra are first decomposed using a Principal Component Analysis (PCA) model containing a user-specified number of principal components or "factors". The resulting factors are then used as the filter.
stored in the indicated files and y-values of 13.2, 19.0, 5.3, and 8.3
(with a text description of the y-values as "concentration"). The first
three files would be used for calibration, the last file for validation.


[[Image:Experimentreadr diagram.png||600px| ]]
====Adding Correction Spectra====


===See Also===
To add correction spectra to the filter, click on the '''Add''' button and select the file type to read (e.g. "Hamilton Sundstrand ASF files") and click OK.


[[analysis]], [[autoimport]], [[xclreadr]]
Browse to the folder containing the file(s) to load, select one or more files (using shift-click or control-click to select multiple files), and click "Open".
 
The files will be loaded and added to the list. You will be advised if the spectral axis of the loaded files appears to be incompatible with the currently loaded x-block data. Any incompatible spectra will not be loaded.
 
[[Image:Cpsa_correction_loaded.png]]
 
====Removing Correction Spectra====
 
Any file that has been loaded as a correction spectrum can be removed from the filter by simply clicking on the filename in the list on the left, and then clicking on the '''Remove''' button. The file will be discarded. Note that after discarding any spectra from the filter, any currently-loaded model will be cleared and have to be recalculated because the filter has now changed and the model is no longer current.
 
====Choosing Number of Factors====
 
The CPSA filter calculates a PCA model from the correction spectra and keeps some number of factors. The edit box on the right of the control specifies the number of factors to use in the filter. The higher the number, the more degrees of freedom the filter will remove (and the increased chance of removing useful signal). In general, a lower number of factors is recommended unless the filter is not performing sufficiently for a given application.
 
====Viewing the Correction Spectra====
 
The selected correction spectra can be viewed in a plot by clicking the '''Plot''' button. The spectra selected on the left will be displayed, but all loaded correction spectra can be viewed using the Plot Controls which will appear along with the plot of the spectra. Regions of these spectra can be excluded using the Select button and Edit/Exclude or Include menu options in the Plot Controls.
 
===Force PC80 Compatible Preprocessing===
 
If this option is unchecked, the user may use any preprocessing available in the custom preprocesing interface. However, the model created will be compatible with Eigenvector Research software such as Solo_Predictor, Solo, and PLS_Toolbox but will not be compatible with the Hamilton Sundstrand PC80 and CPSA32 applications.
 
When checked, the preprocessing selected in the pull-down menu to the right of the option (including: None, Mean Centering, Autoscaling) will be used and any other preprocessing will be removed. This mode assures that the model created will be as compatible as possible with the PC80 and CPSA32 applications.

Revision as of 16:35, 27 August 2009

CPSA is a preprocessing method used to remove interferences from data prior to building a regression model. The CPSA filter includes one or more of the following corrections:

  1. multivariate filtering of components defined by a set of clutter or "correction" spectra.
  2. detrending (aka baselining) of the input data in windows of variables.
  3. an integrated thickness/pathlength correction.

CPSA Panel

The CPSA tools are accessed through the CPSA panel of Analysis (click on the "CPSA Settings" button just below the top status panel). The CPSA settings panel is only available when using the PCR or PLS Analysis methods and only enabled when calibration data has already been loaded. The default settings for the panel are shown below.

Cpsa panel empty.png


Four different options categories are available on the panel:

  • Correction Spectra - Allows selecting correction spectra which describe background components which should be removed from the spectra prior to analysis.
  • Regions/Baselines - Defines regions of the spectra and the baseline order to apply to each region.
  • Thickness Correction - Determines whether or not to use thickness/pathlength correction (requires thickness/pathlength reference in calibration data)
  • Force PC80 Compatible Preprocessing - Determines whether to enforce strict preprocessing rules on the created model.


Correction Spectra

The list on the left of this option shows the file(s) which have been selected by the user as correction spectra (aka background or clutter spectra) which should be removed from all calibration and test spectra. To perform the filter, the correction spectra are first decomposed using a Principal Component Analysis (PCA) model containing a user-specified number of principal components or "factors". The resulting factors are then used as the filter.

Adding Correction Spectra

To add correction spectra to the filter, click on the Add button and select the file type to read (e.g. "Hamilton Sundstrand ASF files") and click OK.

Browse to the folder containing the file(s) to load, select one or more files (using shift-click or control-click to select multiple files), and click "Open".

The files will be loaded and added to the list. You will be advised if the spectral axis of the loaded files appears to be incompatible with the currently loaded x-block data. Any incompatible spectra will not be loaded.

Cpsa correction loaded.png

Removing Correction Spectra

Any file that has been loaded as a correction spectrum can be removed from the filter by simply clicking on the filename in the list on the left, and then clicking on the Remove button. The file will be discarded. Note that after discarding any spectra from the filter, any currently-loaded model will be cleared and have to be recalculated because the filter has now changed and the model is no longer current.

Choosing Number of Factors

The CPSA filter calculates a PCA model from the correction spectra and keeps some number of factors. The edit box on the right of the control specifies the number of factors to use in the filter. The higher the number, the more degrees of freedom the filter will remove (and the increased chance of removing useful signal). In general, a lower number of factors is recommended unless the filter is not performing sufficiently for a given application.

Viewing the Correction Spectra

The selected correction spectra can be viewed in a plot by clicking the Plot button. The spectra selected on the left will be displayed, but all loaded correction spectra can be viewed using the Plot Controls which will appear along with the plot of the spectra. Regions of these spectra can be excluded using the Select button and Edit/Exclude or Include menu options in the Plot Controls.

Force PC80 Compatible Preprocessing

If this option is unchecked, the user may use any preprocessing available in the custom preprocesing interface. However, the model created will be compatible with Eigenvector Research software such as Solo_Predictor, Solo, and PLS_Toolbox but will not be compatible with the Hamilton Sundstrand PC80 and CPSA32 applications.

When checked, the preprocessing selected in the pull-down menu to the right of the option (including: None, Mean Centering, Autoscaling) will be used and any other preprocessing will be removed. This mode assures that the model created will be as compatible as possible with the PC80 and CPSA32 applications.