Diviner
Page under construction
Diviner
Diviner is a semi-automated machine learning (Semi-AutoML) tool specifically designed to enhance the development of multivariate calibration models for linear regression. Unlike traditional AutoML systems that entirely automate the machine learning workflow, often at the expense of domain-specific insights and transparency, Diviner strikes a balance between automation and expert involvement. It allows users to leverage automation efficiently while maintaining control over critical decision points in the modeling process. This hybrid approach addresses key shortcomings of AutoML, such as the lack of domain knowledge, overfitting, and limited customization, by integrating user input to guide model development more effectively.
Diviner is a tool designed for calibrating linear models, specifically Partial Least Squares (PLS) and regularized multiple linear regression (MLR) models, such as Elastic Net. It provides a comprehensive workflow that includes outlier assessment, a grid search for preprocessing methods, variable selection, and user-guided model refinement.
Due to its extensive library of preloaded preprocessing methods, Diviner is particularly suited for chemometrics and spectral data analysis. However, users can also create custom preprocessing libraries, making Diviner a versatile tool for linear regression tasks with various data types.
Data Workflow
Exploratory Module:
Data loading: The process begins with data being loaded into the system. If a test dataset is available, it should be loaded at this time so it can be used to evaluate the performance of the models in that test dataset. Note: if a test set is not loading before the run, applying the models to the test set a posteriori won't be possible. Choice of algorithm: PLS is set by default, and MLR must be activated in the options. While the optimization of the PLS models is part of the initial grid search in diviner, MLR (Elastic Nets) on the other hand, performs an optimization routine to find the best penalty for each model. Preprocessing: This step involves selecting multiple preprocessing methods depending on the data type and application. In the same step, preprocessing methods use for outlier assessment must be selective. Cross-validation (CV): Diviner supports all modes of cross-validation in the PLS_Toolbox and Solo. This step also sets the number of PLS latent variables (LVs) that will be used in the initial grid search. Outlier assessment: Diviner automatically assesses possible outliers in the dataset using a combination of robust PCA and PLS with different preprocessing methods. Auto-Variable Selection: Variable selection in the first module is carry out by using two fast algorithms VIP (Variable Importance in Projection) and sRatio (Selectivity Ratio). https://www.wiki.eigenvector.com/index.php?title=Selectvars
Preliminary Output & First Model Selection: The initial output is generated, and a preliminary model is selected based on performance metrics like RMSECV (Root Mean Square Error of Cross-Validation).
Refinement Module:
• Further Variable Selection: In this step, additional refinement of the variable selection is carried out, often using techniques like interval Partial Least Squares (iPLS) to fine-tune which variables contribute to the model. • Outlier Reinclusion: If any outliers previously excluded are significant upon further analysis, they might be re-evaluated and potentially reintegrated into the models. • Final Model Selection: After all refinements are made, the best-performing model is selected for final output.
Output:
• The process concludes with generating the final model, which is the culmination of the exploratory and refinement processes, ensuring that the model is robust, accurate, and tailored to the specific dataset.
This structured workflow allows for a balanced approach that combines automated processes with critical user feedback.
Bold text
