MCCTTool: Difference between revisions

From Eigenvector Research Documentation Wiki
Jump to navigation Jump to search
imported>Scott
No edit summary
No edit summary
 
(20 intermediate revisions by 4 users not shown)
Line 4: Line 4:
=Introduction=
=Introduction=


No two instruments are identical. Data generated from different instruments can rarely be used with the same model. It may be prohibitively expensive or otherwise impossible to run an entire complement of calibration samples on new instrument. A calibration transfer step can be developed to transform "slave" instrument data to <u>look like</u> it came from a "master" instrument.
No two instruments are identical. Data generated from different instruments can rarely be used with the same model. It may be prohibitively expensive or otherwise impossible to run an entire complement of calibration samples on a new instrument. A calibration transfer step can be developed to transform "secondary" instrument data to <u>look like</u> it came from a "primary" instrument.


Model-centric Calibration Transfer (MCCT) refers to the situation where a model has been developed on a "master" instrument and is the basis of creating a "slave" model that includes a calibration transfer step in the preprocessing structure. It is assumed there data sets used for calibration transfer for each instruments are matched samples.
Model-centric Calibration Transfer (MCCT) refers to the situation where a model has been developed on a "primary" instrument and is the basis of creating a "secondary" model that includes a calibration transfer step in the preprocessing structure. It is assumed data sets used for calibration transfer for each instruments are matched samples (identical samples run on each instrument).


How data is preprocessed can affect the standardization. The location of calibration transfer step within the preprocessing can have a large impact on how well the slave model will perform. The MCCT Tool allows the testing of different combinations of calibrations transfer methods and parameters, and where the calibration transfer step is inserted into the preprocessing.
How data is preprocessed can affect the standardization. The location of calibration transfer step within the preprocessing can have a large impact on how well the secondary model will perform. The MCCT Tool allows the testing of different combinations of calibrations transfer methods and parameters, and where the calibration transfer step is inserted into the preprocessing. The MCCT Tool relies on the [[MCCT Object]] that can be used directly from the Matlab command line or in a function (see the MCCTOBJECTDEMO for an example).
 
NOTE: Creating a calibration transfer model independently can be done with the standard [[Calibration_Transfer | Calibration Transfer]] interface.
 
 
==Supported Model Types==
: '''PCA, PLS, PCR, LWR, CLS, MLR, ANN, SVM'''


=MCCT Tool Interface=
=MCCT Tool Interface=
[[Image:MCCTTool.jpg|500| ]]
==Toolbar Buttons==
==Toolbar Buttons==


Line 19: Line 27:
* [Checkbox] - Open window options.
* [Checkbox] - Open window options.


* [Gears] - Calculate a single slave model with current transfer model and preprocess insert location.
* [Gears] - Calculate a single secondary model with current transfer model and preprocess insert location.


* [Arrows] - Zoom graph to window.
* [Arrows] - Zoom graph to window.
Line 29: Line 37:


* [Table] - Open results table as cell array in variable editor.
* [Table] - Open results table as cell array in variable editor.
==Graph==
==Graph==


The graph has two sections, calibration and validation. To create a new slave model on the calibration data and master model need to be loaded. Data and the master model can be dropped on the various icons to load. Hovering the mouse over an particular item will show a temporary information box with a description of the item. Right-clicking on the icon will show a menu of available actions (e.g., view, save, clear) for the item.
The graph has two sections, calibration and validation. To create a new secondary model on the calibration data and primary model need to be loaded. Data and the primary model can be dropped on the various icons to load. Hovering the mouse over a particular item will show a temporary information box with a description of the item. Right-clicking on the icon will show a menu of available actions (e.g., view, save, clear) for the item.


==Calibration Transfer Panels==
==Calibration Transfer Panels==
Line 39: Line 48:
==Results Table==
==Results Table==


Clicking on a row in the table will make the slave model form that row the "current" model in the graph. The slave model icon can then be used to save/export the model for further review. To sort a column, click on the column to select it and then right-click the header to and select a sort menu item. Results can be cleared from the File menu.
Clicking on a row in the table will make the secondary model from that row the "current" model in the graph. The secondary model icon can then be used to save/export the model for further review. To sort a column, click on the column to select it and then right-click the header to and select a sort menu item. Results can be cleared from the File menu.


=Working With MCCTTool=
=Working With MCCTTool=
==Getting Started==
==Getting Started==


The goal of the tool is to create and test multiple calibration transfer scenarios for a given "master" model. Each calibration transfer scenario consists of:
The objective of the tool is to create and test multiple calibration transfer scenarios for a given primary model and handle all necessary bookkeeping operations. Required items:
 
* Primary Data
* Primary Model
* Secondary Data
 
 
Each calibration transfer scenario consists of:


* A calibration transfer model (see [[caltransfer]]) as a preprocessing step.
* A calibration transfer model (see [[caltransfer]]) as a preprocessing step.
* Insert location in the preprocessing steps of the master model.
* Insert location in the preprocessing steps of the primary model.
A given calibration method (DS, PDS, PWDS, SST) may have one or more parameters to "tune" for a given instrument. For example, window size in the case of PDS. The ultimate goal is to produce a new "slave" model based on the "master" model with a calibration transfer preprocessing step inserted to account for instrument variation.
A given calibration method (DS, PDS, DWPDS, SST) may have one or more parameters to "tune" for a given instrument. For example, window size in the case of PDS. The ultimate goal is to produce a new "secondary" model based on the "primary" model with a calibration transfer preprocessing step inserted to account for instrument-to-instrument variation and provide improved prediction metrics.
 
==Adding Data or Model==
==Adding Data or Model==


Line 64: Line 81:
* 1 - After derivative.
* 1 - After derivative.
* 2 - After GLSW.
* 2 - After GLSW.
Note: By default, the calibration transfer models automatically created will include the preprocessing steps prior to the insert point. With the example above and the insert location at 1, the slave model will have the following preprocessing steps:
Note: By default, the calibration transfer models automatically created will include the preprocessing steps prior to the insert point. With the example above and the insert location at 1, the secondary model will have the following preprocessing steps:
* Calibration Transfer model (derivative preprocessing embedded)
* Calibration Transfer model (derivative preprocessing embedded)
* GLSW
* GLSW
* Mean Center
* Mean Center


At the insert point, slave data is processed to "look like" it was measured on the master instrument.  Preprocessing steps in the master model prior to the insert point are copied by default into the calibration transfer model object.  Customization is possible by editing in the calibration transfer GUI (see caltransfergui).  Preprocessing steps in the slave model after the calibration transfer model are copied and not changeable.
At the insert point, secondary data is processed to "look like" it was measured on the primary instrument.  Preprocessing steps in the primary model prior to the insert point are copied by default into the calibration transfer model object.  Customization is possible by editing in the calibration transfer GUI (see caltransfergui).  Preprocessing steps in the secondary model after the calibration transfer model are copied and not changeable.


==Interpreting Results==
==Interpreting Results==
Use the calibration and validation difference ratios for a general guide to performance on the x-block. Smaller ratios indicate that the slave data - post application of the calibration transfer model and all preprocessing - is similar to the master data. '''<need equation here?>'''
Use the calibration and validation difference ratios (column headings '''''Cal Diff Data''''' and '''''Val Diff Data''''') for a general guide to performance on the x-block. Smaller ratios indicate that the secondary data - post application of the calibration transfer model and all preprocessing - is similar to the primary data. The toolbox function [[calcdifference]] is used to compute the ratios.


For regression models (that use a y-block) various RMSE values are calculated. Please note that at this time only regression models with univariate Y are supported.  The abbreviations for RMSE comparisons are:
For regression models (that use a y-block) various RMSE values are calculated. Please note that at this time only regression models with univariate Y are supported.  The abbreviations for RMSE comparisons are:


* CalM - Master Model Prediction of Master Calibration Data
* CalM - Primary Model Prediction of Primary Calibration Data
* CalS - Slave Model Prediction of Slave Calibration Data
* CalS - Secondary Model Prediction of Secondary Calibration Data
* CalY - Calibration Y Data
* CalY - Calibration Y Data


* ValM - Master Model Prediction of Master Validation Data
* ValM - Primary Model Prediction of Primary Validation Data
* ValS - Slave Model Prediction of Slave Validation Data
* ValS - Secondary Model Prediction of Secondary Validation Data
* ValY - Validation Y Data
* ValY - Validation Y Data


For single block methods (PCA) the RMSE(CalM,CalS) and RMSE(ValM,CalS) column become a difference summary based on the following calculations:
For single block methods (PCA) the columns '''''Cal Diff Scores''''' and '''''Val Diff Scores'''''  are difference ratios that use
 
T<sub>m</sub> = scores from primary (either cal or val)


T<sub>m</sub> = scores from master
T<sub>s</sub> = scores from secondary (either cal or val)
T<sub>s</sub> = scores from slave


m = number of samples
and the toolbox function [[calcdifference]].


k = number of principal components
NOTE: Clicking a column then right-clicking the header will display a menu where sorting is available.


RMSE = <math>\sqrt{\frac{\sum\sum((T_m-T_s).^{\wedge}2)}{m\,*\,k}}</math>


NOTE: Clicking a column then right-clicking the header will display a menu where sorting is available.
=Model_Exporter and MCCT models=
Calibration transfer models created by this MCCT are exportable by Model_Exporter if the master model is exportable. The only difference is that the MCCT-generated model has an additional preprocessing step which itself is exportable.

Latest revision as of 10:35, 21 March 2023

Model-centric Calibration Transfer Tool

Introduction

No two instruments are identical. Data generated from different instruments can rarely be used with the same model. It may be prohibitively expensive or otherwise impossible to run an entire complement of calibration samples on a new instrument. A calibration transfer step can be developed to transform "secondary" instrument data to look like it came from a "primary" instrument.

Model-centric Calibration Transfer (MCCT) refers to the situation where a model has been developed on a "primary" instrument and is the basis of creating a "secondary" model that includes a calibration transfer step in the preprocessing structure. It is assumed data sets used for calibration transfer for each instruments are matched samples (identical samples run on each instrument).

How data is preprocessed can affect the standardization. The location of calibration transfer step within the preprocessing can have a large impact on how well the secondary model will perform. The MCCT Tool allows the testing of different combinations of calibrations transfer methods and parameters, and where the calibration transfer step is inserted into the preprocessing. The MCCT Tool relies on the MCCT Object that can be used directly from the Matlab command line or in a function (see the MCCTOBJECTDEMO for an example).

NOTE: Creating a calibration transfer model independently can be done with the standard Calibration Transfer interface.


Supported Model Types

PCA, PLS, PCR, LWR, CLS, MLR, ANN, SVM

MCCT Tool Interface

MCCTTool.jpg

Toolbar Buttons

Buttons from left to right:

  • [Lambda] - Open the Workspace browser.
  • [Checkbox] - Open window options.
  • [Gears] - Calculate a single secondary model with current transfer model and preprocess insert location.
  • [Arrows] - Zoom graph to window.
  • [Plus Magnify] - Zoom in graph.
  • [Minus Magnify] - Zoom out graph.
  • [One to One] - Zoom to default.
  • [Table] - Open results table as cell array in variable editor.

Graph

The graph has two sections, calibration and validation. To create a new secondary model on the calibration data and primary model need to be loaded. Data and the primary model can be dropped on the various icons to load. Hovering the mouse over a particular item will show a temporary information box with a description of the item. Right-clicking on the icon will show a menu of available actions (e.g., view, save, clear) for the item.

Calibration Transfer Panels

These panels are used to create combinations of calibration transfer scenarios to try. One or more calibration transfer insert locations can be set via the Prepro Insert button. Models are saved during the calculation so if various combinations are changed resulting in duplicate scenarios, the model will be retrieved from saved results rather than recalculated.

Results Table

Clicking on a row in the table will make the secondary model from that row the "current" model in the graph. The secondary model icon can then be used to save/export the model for further review. To sort a column, click on the column to select it and then right-click the header to and select a sort menu item. Results can be cleared from the File menu.

Working With MCCTTool

Getting Started

The objective of the tool is to create and test multiple calibration transfer scenarios for a given primary model and handle all necessary bookkeeping operations. Required items:

  • Primary Data
  • Primary Model
  • Secondary Data


Each calibration transfer scenario consists of:

  • A calibration transfer model (see caltransfer) as a preprocessing step.
  • Insert location in the preprocessing steps of the primary model.

A given calibration method (DS, PDS, DWPDS, SST) may have one or more parameters to "tune" for a given instrument. For example, window size in the case of PDS. The ultimate goal is to produce a new "secondary" model based on the "primary" model with a calibration transfer preprocessing step inserted to account for instrument-to-instrument variation and provide improved prediction metrics.

Adding Data or Model

To add data or a model to the MCCT Tool, drag and drop it from the Workspace Browser, Matlab Workspace, or from a file. The data will automatically be loaded into the tool. The File menu can also be used to load items.

Calibration Transfer Methods

Select the methods to try and the combinations of parameters to use. It may be useful to choose large "step" sizes when surveying over a large range of values. PDS and DWPDS step sizes are odd.

Preprocessing Insert

Choose what positions the calibrations transfer step will be inserted at. More than one value can be used. A value of 0 indicates the transfer step will occur before all preprocessing. For example, if a model has 3 preprocessing steps ('Derivative','GLS Weighting','Mean Center') then:

  • 0 - Before all preprocessing.
  • 1 - After derivative.
  • 2 - After GLSW.

Note: By default, the calibration transfer models automatically created will include the preprocessing steps prior to the insert point. With the example above and the insert location at 1, the secondary model will have the following preprocessing steps:

  • Calibration Transfer model (derivative preprocessing embedded)
  • GLSW
  • Mean Center

At the insert point, secondary data is processed to "look like" it was measured on the primary instrument. Preprocessing steps in the primary model prior to the insert point are copied by default into the calibration transfer model object. Customization is possible by editing in the calibration transfer GUI (see caltransfergui). Preprocessing steps in the secondary model after the calibration transfer model are copied and not changeable.

Interpreting Results

Use the calibration and validation difference ratios (column headings Cal Diff Data and Val Diff Data) for a general guide to performance on the x-block. Smaller ratios indicate that the secondary data - post application of the calibration transfer model and all preprocessing - is similar to the primary data. The toolbox function calcdifference is used to compute the ratios.

For regression models (that use a y-block) various RMSE values are calculated. Please note that at this time only regression models with univariate Y are supported. The abbreviations for RMSE comparisons are:

  • CalM - Primary Model Prediction of Primary Calibration Data
  • CalS - Secondary Model Prediction of Secondary Calibration Data
  • CalY - Calibration Y Data
  • ValM - Primary Model Prediction of Primary Validation Data
  • ValS - Secondary Model Prediction of Secondary Validation Data
  • ValY - Validation Y Data

For single block methods (PCA) the columns Cal Diff Scores and Val Diff Scores are difference ratios that use

Tm = scores from primary (either cal or val)

Ts = scores from secondary (either cal or val)

and the toolbox function calcdifference.

NOTE: Clicking a column then right-clicking the header will display a menu where sorting is available.


Model_Exporter and MCCT models

Calibration transfer models created by this MCCT are exportable by Model_Exporter if the master model is exportable. The only difference is that the MCCT-generated model has an additional preprocessing step which itself is exportable.