Diviner
Page under construction
Diviner
Diviner is a semi-automated machine learning (Semi-AutoML) tool specifically designed to enhance the development of multivariate calibration models for linear regression. Unlike traditional AutoML systems that entirely automate the machine learning workflow, often at the expense of domain-specific insights and transparency, Diviner strikes a balance between automation and expert involvement. It allows users to leverage automation efficiently while maintaining control over critical decision points in the modeling process. This hybrid approach addresses key shortcomings of AutoML, such as the lack of domain knowledge, overfitting, and limited customization, by integrating user input to guide model development more effectively.
Diviner is a tool designed for calibrating linear models, specifically Partial Least Squares (PLS) and regularized multiple linear regression (MLR) models, such as Elastic Net. It provides a comprehensive workflow that includes outlier assessment, a grid search for preprocessing methods, variable selection, and user-guided model refinement.
Due to its extensive library of preloaded preprocessing methods, Diviner is particularly suited for chemometrics and spectral data analysis. However, users can also create custom preprocessing libraries, making Diviner a versatile tool for linear regression tasks with various data types.
Data Workflow
Exploratory Module:
• Data loading and Cross-Validation (CV): The process begins with data being loaded into the system. If a test set is available, it should be loaded at this time as well since it is the only way to use the test Cross-validation (CV) is applied to ensure the reliability of the model’s performance by repeatedly splitting the data into training and testing subsets. • Preprocessing: This step involves preparing the data by applying various transformations and normalizations to improve model performance. This can include operations like scaling, centering, or applying derivative techniques. • Outlier Auto-Detection: Outliers are automatically detected to identify and potentially exclude data points that could skew the model’s accuracy. • Auto-Variable Selection: This automated step selects the most relevant variables from the dataset, reducing dimensionality and improving the model’s focus on significant predictors. • Partial Least Squares (PLS) and regression-based multiple linear regression (Reg-MLR): These regression techniques generate initial models to provide preliminary insights into the relationships within the data. • Preliminary Output & First Model Selection: The initial output is generated, and a preliminary model is selected based on performance metrics like RMSECV (Root Mean Square Error of Cross-Validation).
Refinement Module:
• Further Variable Selection: In this step, additional refinement of the variable selection is carried out, often using techniques like interval Partial Least Squares (iPLS) to fine-tune which variables contribute to the model. • Outlier Reinclusion: If any outliers previously excluded are significant upon further analysis, they might be re-evaluated and potentially reintegrated into the models. • Final Model Selection: After all refinements are made, the best-performing model is selected for final output.
Output:
• The process concludes with generating the final model, which is the culmination of the exploratory and refinement processes, ensuring that the model is robust, accurate, and tailored to the specific dataset.
This structured workflow allows for a balanced approach that combines automated processes with critical user feedback.
Bold text
