Model data preprocessing
Preprocessing saves a sequence of spectral transforms with the model configuration. Clarity applies the configured processing consistently when preparing model inputs for training, validation, testing, and inference.
To configure it, open a draft model version and select Overview > Advanced Settings > Preprocessing.
Available processors
| Processor | Methods |
|---|---|
| Band processor | None, Spectral Binning, Spectral Band Filtering |
| Scaling | None, Min-Max, Center, Scale, Standardize |
| Normalization | None, L1, L2, Max, Area |
| Scatter correction | None, MSC, SNV |
| Dimension reduction | None, Principal Component Analysis |
| Absorbance | None, Absorbance |
| Smoothing | None, Detrend, Derivative, Smooth, Convex Hull Removal |
All processors default to None.
Spectral Binning offers downsampling factors of 1, 2, 4, 8, and 16. Spectral Band Filtering lets you keep configured band or wavelength ranges. Smooth can use an Average or Median algorithm.
Model-level Principal Component Analysis reuses the projection stored when the selected dataset version was finalized with Perform PCA; it does not fit a new PCA. If the dataset did not generate PCA artifacts, model validation fails.
The current validator rejects PCA combined with band processing, absorbance, smoothing, normalization, or scaling. Scatter correction with MSC or SNV is currently permitted. The model form reports incompatible combinations before training.
Choosing transforms
Use a transform only for a specific, understood data issue:
- Band filtering can remove unusable ranges, but it also removes information.
- Scaling and normalization change magnitude relationships.
- MSC and SNV address scatter effects under different assumptions.
- PCA reduces dimensionality but makes inputs less directly interpretable.
- Derivatives and smoothing can suppress baselines or noise while also changing narrow features.
Fit every learned preprocessing parameter on Training data only. Use Validation data to select a transform, then evaluate the locked pipeline on untouched Test data. A transform that improves one sensor or acquisition condition can reduce transfer to another.