Data Editing Scaling and Preprocessing and Release Notes Version 7 0 2: Difference between pages

From Eigenvector Research Documentation Wiki
(Difference between pages)
Jump to navigation Jump to search
imported>Jeremy
(Importing text file)
 
imported>Jeremy
No edit summary
 
Line 1: Line 1:
===Data Editing, Scaling, and Preprocessing===
==Changes and Bug Fixes in Version 7.0.2==
:[[preprocess]] - Selection and application of standard preprocessing structures.
 
:[[preprouser]] - User-defined preprocessing methods.
===Bug Fixes and Enhancements===
:[[editds]] - Editor for DataSet Objects.
{|
 
|----valign="top"
====Baselining and Offset Correction====
|'''[[analysis]]'''
:[[baseline]] - Subtracts a polynomial baseline offset from spectra.
|
:[[baselinew]] - Baseline using windowed polynomial filter.
* Allow split cal/val even when no cal is present
:[[wlsbaseline]] - Weighted least squares baseline function.
* Fix for error when loading old model with custom cross-validation (loaded cvi which had only the INCLUDED samples liseted. New detail.cvi field contains both included and excluded samples and is what crossval was expecting to get)
:[[mscorr]] - Multiplicative scatter/signal correction (MSC).
* Fix for missing "block" information when drilling down from summary contributions to full contributions in MPCA model
:[[savgol]] - Savitzky-Golay smoothing and differentiation.
* Allow relative T and Q contributions in MPCA models
* Fix for multiway bug in calculating Q contributions
* Give warning when user attempts to change conf. limit on batch maturity model type that this has no effect on shown conf. limits.
:[[lsq2top]] - Fits a polynomial to the top/(bottom) of data.
* Show used conf. limit in plot controls for Batch Maturity
:[[med2top]] - Fits a constant to top/(bottom) of data.
 
====Derivatives and Smoothing====
 
:[[savgol]] - Savitzky-Golay smoothing and differentiation.
|----valign="top"
:[[polyinterp]] - Polynomial interpolation, smoothing, and differentiation.
|'''[[batchfold]]'''
====Sample-wise scaling corrections (Normalizing)====
'''[[bspcgui|Batch Processor]]'''
:[[mscorr]] - Multiplicative scatter/signal correction (MSC).
|
:[[normaliz]] - Normalize rows of matrix.
* If steps are disabled, ignore extraction by steps!
:[[snv]] - Standard normal variate scaling.
* Remove forced removal of steps if Batch Maturity.
====Variable-wise scaling and offset corrections====
* Add name to dataset.
:[[auto]] - Autoscales matrix to mean zero unit variance.
* Add per batch linear axis scale.
:[[mncn]] - Scale matrix to mean zero.
* Updates for alignment on BM and other.
:[[medcn]] - Median center scales matrix to median zero.
* Fix model saving. Fix cow options. Add 'none' option in alignment. Add better loading of model and settings. Fix tab enable on load of model.
:[[logdecay]] - Mean centers and variance scales a matrix using the log decay of the variable axis.
* Fix for allowing no steps. Become all one step.
:[[scale]] - Scales data using specified means and std. devs.
* Add new plotting style, apply to new data, and remove class 0 from batch list.
:[[rescale]] - Scales data back to original scaling.
* Always push data into the same Analysis window (if it is still open), otherwise use a new window
:[[gscale]] - Group/block scaling for a single or multiple blocks.
* If model or data is loaded, ask how to load data when pushed (calibration / validation)
:[[gscaler]] - Applies group/block scaling to submatrices of a single matrix.
* Add default alignment plus default method for BM and other.
====Interference Removal and Missing Data====
* Add "stacked" plotting on batch plot.
:[[glsw]] - Generalized least-squares weighting/preprocessing.
* Update to drag patch behavior in linear view.
:[[mdcheck]] - Missing Data Checker and infiller.
* Fix for batch list selections, make default batch plot style = stack.
:[[osccalc]] - Calculates orthogonal signal correction (OSC).
* Remove unneeded batch selection now that Class 0 has been removed.
====Sample and Variable handling====
 
:[[coadd]] - Reduce resolution through combination of adjacent variables or samples.
|----valign="top"
:[[delsamps]] - Deletes samples (rows) or variables (columns) from data matrices.
|'''[[b3spline]]'''
:[[excludemissing]] - Automatically exclude too-much missing data in a matrix.
|
:[[lamsel]] - Determines indices of wavelength axes in specified ranges.
* Fix error in display option handling
:[[shuffle]] - Randomly re-orders matrix and multiple blocks rows.
 
:[[specedit]] - GUI for selecting spectral regions on a plot.
|----valign="top"
|'''[[batchmaturity]]'''
|
:[[unfoldm]] - Rearranges (unfolds) an augmented matrix to row vectors.
* Added asymmetric standard deviation as method to calculate confidence limits
:[[unfoldmw]] - Unfolds multiway arrays along specified order.
* Added confidence limit algorithm (clalgorithm) option with default to asymmetric least squares (astd)
:[[batchdigester]] - Parse wafer or batch data into MPCA or Summary PCA form.
* Adjusted default confidence limit to 95% to match default in other level 2 functions
* Remove weighting applied to deviations when calculating the score limits using "percentile" method
* Don't calculate score limits when building raw model as this would be done unnecessarily for 10 PCs. This could be time consuming.
 
|----valign="top"
|'''[[boxplot]]'''
|
* No "Extreme" outliers plotted if there were no "Standard" outliers. This was the case for either upper or lower outliers, so upper (lower) extremes only plotted if there were upper (lower) standard outliers.
 
|----valign="top"
|'''[[browse]]'''
|
* Add message saying browse is initializing
 
|----valign="top"
|'''[[corrspecgui]]'''
|
* Fix typo in plot type.
 
|----valign="top"
|'''[[summary]]'''
|
* Fix for error when all of a given variable are excluded/missing
 
|----valign="top"
|'''[[experimentreadr]]'''
|
* Switch cal/val class numbers (so calibration is 0 and shows as black circles, and 1 as red triangles as with scores plots)
* Handle case when all samples are converted to validation
 
|----valign="top"
|'''[[genalgplot]]'''
|
* add drawnow to make sure some plots get updated when we switch from selection plot to the information plot
 
|----valign="top"
|'''[[modelcache]]'''
|
* Add new deletedates mode to modelcache
 
|----valign="top"
|'''[[mscorr]]'''
|
* Fix typo in error message
 
|----valign="top"
|'''[[parafac]]'''
|
* Fix for serious but rare bug in PARAFAC: For higher than three-way, the constraint in mode two was also imposed in mode three. So the bug is only seen when those constraints are different. Most of the time constraints would just be nonneg all over the place,so bug is unlikely to be seen.
 
|----valign="top"
|'''[[peakfind]]'''
|
* Don't do search for peaks if fewer than window*2 variables!
 
|----valign="top"
|'''[[plotgui|Plot Controls]]'''
|
* Add separators above Bar and Mesh to make menu easier to read
* Add "enhanced surface" mode
* Better handling duplication of data as needed for 3D plots (to avoid errors when plotting)
* Change settings on viewinterpolated so it will be available from the settings control button on the toolbar
* Fix for plotting scatter plots with n-way data in 3rd dimension (xdata is row vector instead of column vector)
* Don't reset 'PlotBoxAspectRatioMode','CameraViewAngleMode', or 'DataAspectRatioMode' in 2008b or later (seems to cause strange plot box resizing problems)
* Better position labels when rotated text is being used
* Add ability to use logical in search
 
|----valign="top"
|'''Adjust Axis Limits Interface'''
|
* Fix use with multiple axes and multiple figures. Fix bugs with initializing settings. Better handle restoring color.
* Fix for color of background when target figure has BLACK (or dark gray) background (can't see text!!)
 
|----valign="top"
|'''[[plsda]]'''
|
* Treat "0" as unknown class only if input y has more than 2 unique values
 
|----valign="top"
|'''[[preprocess]]'''
|
* Add "Favorites" button to
: (a) move certain methods to the top of the preprocessing list OR
: (b) to create new aggregate methods from the current selection of multiple methods
* Add "Hide/Unhide" button to hide items you don't use often
* Add hidden support for font size changing
 
|----valign="top"
|'''[[splitcaltest]]'''
|
* Fix bug where splitcaltest does nothing (all samples remain as calibration) if input data is "short and wide", as with nir_data for example with SVM, or when ncomp >=10 for PCA, LWR, etc.
* Remove requirement that the input data were acquired in a random order
* Initial demo added
 
|----valign="top"
|'''[[tconcalc]]'''
|
* Add support for tcon calculation from PCR and PLS models even when tconcalc is passed ONLY the prediction structure (as long as the necessary eigenvalues information is in the model details)
 
|----valign="top"
|'''[[trendtool]]'''
|
* Consider a "viewSpec" request for the a spectrum beyond the highest numbered spectrum as a request for "the last" spectrum (e.g. "inf" will give the max)
* Add 'interpolation' as new property that trendtool can set on the trend view
* Add ability to access this through evrigui as property: obj.setInterpolation(n)
* Add plottype surface and evrigui connection to modify it (setPlottype)
 
|----valign="top"
|'''[[EVRIGUI Objects]]'''
|
* Add fieldnames to EVRIGUI object to allow tab-completion of valid methods and properties
 
|----valign="top"
|'''[[EVRIModel Objects]]'''
|
* Rearrange logic when updating from old model version (generalize copying of fields from old model into new one
* Add conrearrange as private method to re-arrange contributions into "used", "passed", or "full" forms (like with Solo_Predictor)
* Add "contributions" and "matchvarsmap" (hidden) properties
* Fix logic which assigns calibrate.options.plots and calibrate.options.display settings (also set in top-level)
* Add "matchvars" property to models as option to DISABLE call to matchvars during apply, xhat and tcon/qcon calculations.
* If user turns off model object, don't expect evrimodelversion field (use modelversion only) and automatically extract model contents. Now users can automatically down-grade models using simply:
setplspref('evrimodel','noobject',1)
:then loading the new model
 
|----valign="top"
|'''add3dlight'''
|
* Add "add3dlight" as new GUI utility to add 3D lighting effects for enhanced surface plots
 
|----valign="top"
|'''modelviewertool'''
|
* Fixed a bug in Tucker where the core was plotted as a loading in modelviewer when fitting e.g. Tucker(X,[3 3 1])
 
|----valign="top"
|'''peakfindgui'''
|
* Allow for more or less adjustability in sensitivity depending on the # of variables
* Encode logic to handle non-integer values for found peak position (in case center of mass calculation is used and non-integer peak positions values get returned)
 
|----valign="top"
|'''[[piconnectgui]]'''
|
* better handling of errors thrown during initialization
|----
|}

Revision as of 16:37, 20 November 2012

Changes and Bug Fixes in Version 7.0.2

Bug Fixes and Enhancements

analysis
  • Allow split cal/val even when no cal is present
  • Fix for error when loading old model with custom cross-validation (loaded cvi which had only the INCLUDED samples liseted. New detail.cvi field contains both included and excluded samples and is what crossval was expecting to get)
  • Fix for missing "block" information when drilling down from summary contributions to full contributions in MPCA model
  • Allow relative T and Q contributions in MPCA models
  • Fix for multiway bug in calculating Q contributions
  • Give warning when user attempts to change conf. limit on batch maturity model type that this has no effect on shown conf. limits.
  • Show used conf. limit in plot controls for Batch Maturity


batchfold

Batch Processor

  • If steps are disabled, ignore extraction by steps!
  • Remove forced removal of steps if Batch Maturity.
  • Add name to dataset.
  • Add per batch linear axis scale.
  • Updates for alignment on BM and other.
  • Fix model saving. Fix cow options. Add 'none' option in alignment. Add better loading of model and settings. Fix tab enable on load of model.
  • Fix for allowing no steps. Become all one step.
  • Add new plotting style, apply to new data, and remove class 0 from batch list.
  • Always push data into the same Analysis window (if it is still open), otherwise use a new window
  • If model or data is loaded, ask how to load data when pushed (calibration / validation)
  • Add default alignment plus default method for BM and other.
  • Add "stacked" plotting on batch plot.
  • Update to drag patch behavior in linear view.
  • Fix for batch list selections, make default batch plot style = stack.
  • Remove unneeded batch selection now that Class 0 has been removed.
b3spline
  • Fix error in display option handling
batchmaturity
  • Added asymmetric standard deviation as method to calculate confidence limits
  • Added confidence limit algorithm (clalgorithm) option with default to asymmetric least squares (astd)
  • Adjusted default confidence limit to 95% to match default in other level 2 functions
  • Remove weighting applied to deviations when calculating the score limits using "percentile" method
  • Don't calculate score limits when building raw model as this would be done unnecessarily for 10 PCs. This could be time consuming.
boxplot
  • No "Extreme" outliers plotted if there were no "Standard" outliers. This was the case for either upper or lower outliers, so upper (lower) extremes only plotted if there were upper (lower) standard outliers.
browse
  • Add message saying browse is initializing
corrspecgui
  • Fix typo in plot type.
summary
  • Fix for error when all of a given variable are excluded/missing
experimentreadr
  • Switch cal/val class numbers (so calibration is 0 and shows as black circles, and 1 as red triangles as with scores plots)
  • Handle case when all samples are converted to validation
genalgplot
  • add drawnow to make sure some plots get updated when we switch from selection plot to the information plot
modelcache
  • Add new deletedates mode to modelcache
mscorr
  • Fix typo in error message
parafac
  • Fix for serious but rare bug in PARAFAC: For higher than three-way, the constraint in mode two was also imposed in mode three. So the bug is only seen when those constraints are different. Most of the time constraints would just be nonneg all over the place,so bug is unlikely to be seen.
peakfind
  • Don't do search for peaks if fewer than window*2 variables!
Plot Controls
  • Add separators above Bar and Mesh to make menu easier to read
  • Add "enhanced surface" mode
  • Better handling duplication of data as needed for 3D plots (to avoid errors when plotting)
  • Change settings on viewinterpolated so it will be available from the settings control button on the toolbar
  • Fix for plotting scatter plots with n-way data in 3rd dimension (xdata is row vector instead of column vector)
  • Don't reset 'PlotBoxAspectRatioMode','CameraViewAngleMode', or 'DataAspectRatioMode' in 2008b or later (seems to cause strange plot box resizing problems)
  • Better position labels when rotated text is being used
  • Add ability to use logical in search
Adjust Axis Limits Interface
  • Fix use with multiple axes and multiple figures. Fix bugs with initializing settings. Better handle restoring color.
  • Fix for color of background when target figure has BLACK (or dark gray) background (can't see text!!)
plsda
  • Treat "0" as unknown class only if input y has more than 2 unique values
preprocess
  • Add "Favorites" button to
(a) move certain methods to the top of the preprocessing list OR
(b) to create new aggregate methods from the current selection of multiple methods
  • Add "Hide/Unhide" button to hide items you don't use often
  • Add hidden support for font size changing
splitcaltest
  • Fix bug where splitcaltest does nothing (all samples remain as calibration) if input data is "short and wide", as with nir_data for example with SVM, or when ncomp >=10 for PCA, LWR, etc.
  • Remove requirement that the input data were acquired in a random order
  • Initial demo added
tconcalc
  • Add support for tcon calculation from PCR and PLS models even when tconcalc is passed ONLY the prediction structure (as long as the necessary eigenvalues information is in the model details)
trendtool
  • Consider a "viewSpec" request for the a spectrum beyond the highest numbered spectrum as a request for "the last" spectrum (e.g. "inf" will give the max)
  • Add 'interpolation' as new property that trendtool can set on the trend view
  • Add ability to access this through evrigui as property: obj.setInterpolation(n)
  • Add plottype surface and evrigui connection to modify it (setPlottype)
EVRIGUI Objects
  • Add fieldnames to EVRIGUI object to allow tab-completion of valid methods and properties
EVRIModel Objects
  • Rearrange logic when updating from old model version (generalize copying of fields from old model into new one
  • Add conrearrange as private method to re-arrange contributions into "used", "passed", or "full" forms (like with Solo_Predictor)
  • Add "contributions" and "matchvarsmap" (hidden) properties
  • Fix logic which assigns calibrate.options.plots and calibrate.options.display settings (also set in top-level)
  • Add "matchvars" property to models as option to DISABLE call to matchvars during apply, xhat and tcon/qcon calculations.
  • If user turns off model object, don't expect evrimodelversion field (use modelversion only) and automatically extract model contents. Now users can automatically down-grade models using simply:
setplspref('evrimodel','noobject',1)
then loading the new model
add3dlight
  • Add "add3dlight" as new GUI utility to add 3D lighting effects for enhanced surface plots
modelviewertool
  • Fixed a bug in Tucker where the core was plotted as a loading in modelviewer when fitting e.g. Tucker(X,[3 3 1])
peakfindgui
  • Allow for more or less adjustability in sensitivity depending on the # of variables
  • Encode logic to handle non-integer values for found peak position (in case center of mass calculation is used and non-integer peak positions values get returned)
piconnectgui
  • better handling of errors thrown during initialization