Why Do NIR Outliers Appear — and When Should You Remove Them?

A grain elevator I visited was running protein predictions on incoming wheat. One load came back at 8.2% protein — nearly four points below anything else that day. The operator flagged it, re-scanned, got the same result. Was the instrument lying, or did they actually receive an off-spec load? That question — outlier or real observation? — is one of the most consequential decisions your lab makes. Get it wrong in either direction, and you're either accepting product you should reject, or discarding calibration data that was telling you something true.

What Causes NIR Outliers to Appear?

Outliers come from a handful of distinct sources, and knowing which one you're dealing with changes everything about how you respond. Common causes include measurement errors, instrument drift, improper sample presentation, and genuine extreme variability within a batch. During plant visits I've observed that many outliers trace back to something physical — a dirty sample cup, a poorly filled cell, or a sample that wasn't equilibrated to room temperature before scanning.

Measurement errors happen when something goes wrong in the data collection process itself. In flour milling, a partially filled sample cup or a cracked cell window can produce a spectrum that looks nothing like flour — and your software flags it hard. That's not the flour's fault. That's a process problem, and it's correctable.

Natural variability is a different animal entirely. Sometimes the outlier is a genuine extreme observation — a real sample that sits at the edge of what your calibration has seen before. In a dairy intake setting, milk from a new supplier with unusually high fat content might score as a spectral outlier simply because your calibration was built on a narrower range. That's not an error. That's your model telling you it's operating outside its training experience.

Think of it this way: a calibration model is like a seasoned grain buyer who knows the local market cold. Hand them a sample from three states away with a completely different variety profile, and they'll hesitate — not because they're wrong, but because they haven't seen that before. The hesitation is the signal.

Field tip: Regular calibration checks can significantly cut down on instrument-related outliers. Build a quick check-standard routine into your morning startup — five minutes now prevents a half-day of bad data later.

When Should You Remove or Keep Outliers?

Quality managers often ask me: "Should we delete all outliers?" The answer depends entirely on what caused them. If the outlier traces back to a measurement error — dirty optics, wrong sample depth, instrument drift — remove it. It doesn't represent the sample. It represents a lab process failure.

If the outlier is real — if that wheat load genuinely came in at 8.2% protein — removing it from your calibration set would be a mistake. Your model needs to know that extreme exists. In animal feed production, an outlier in a raw ingredient's amino acid profile might be the first sign of an adulteration event. Deleting it means you'll miss the next one.

Watch out: Removing genuine outliers from your calibration data narrows your model's effective range and trains it to ignore the exact conditions it most needs to handle.

In some cases, a cluster of outliers at grain receiving — all pointing to elevated moisture on loads from a specific supplier — is a process signal, not a data artifact. Three consecutive loads from the same origin showing moisture above 16% when your spec is 14% is telling you something about that supplier's drying operation. Keep those points. Document them. Use them.

How to Detect and Handle Outliers in NIR Data

Detection starts with your software's built-in diagnostics. Most NIR platforms report a Mahalanobis distance (GH or H-statistic) and a spectral residual (T-statistic). The GH distance tells you whether your sample's composition falls outside the calibration population — a compositional outlier. The T-statistic tells you whether the spectrum itself looks physically abnormal — a spectral outlier. You need to check both, because they're catching different problems.

Beyond the software flags, statistical tools like the interquartile range (IQR) and standard deviations (SD) help you spot extreme values in your reference data before they ever enter your model. In my training sessions, I walk labs through running these checks on their wet chemistry reference sets first — because if your Kjeldahl values have a data-entry error at 32.4% protein in a soybean meal dataset, that outlier will wreck your RMSEC and you won't know why.

Formal statistical tests like Grubbs' test or Dixon's Q test give you a defensible, documented basis for your removal decisions. That matters when your auditors ask why you excluded a point. "The GH exceeded 3.0 and the wet chemistry value was confirmed as a transcription error" is an answer. "It looked weird" is not.

Note: Review your calibration models regularly — not just when predictions drift. An outlier that seemed minor six months ago can compound over time and pull your model's slope off by enough to affect pricing decisions at a grain elevator or nutrient guarantees at a feed mill.

One thing I see labs skip: re-scanning the flagged sample before making any decision. Before you classify an outlier as real or artifactual, present the sample again under controlled conditions — correct temperature, clean cell, proper fill. If the second scan produces the same spectrum, you have a consistent result. If it produces a normal spectrum, your first scan was the problem. That one extra scan saves you from making the wrong call.

Practical Steps for Managing NIR Outliers

1. 1Check Instrument Calibration — Verify your NIR instrument against a certified check standard before the shift starts. Instrument drift is a far more common outlier source than most labs realize.
2. 2Identify Outliers — Use your software's GH and T-statistics alongside IQR and SD checks on your reference data to pinpoint extreme observations in your dataset.
3. 3Assess the Cause — Re-scan the flagged sample under proper conditions. Determine whether the outlier comes from a measurement problem or represents genuine sample variability.
4. 4Decide on Action — Remove outliers confirmed as measurement errors and document your reasoning. Keep genuine extreme observations — they protect your calibration's range.
5. 5Update Calibration Models — After any removal or retention decision, recheck your RMSECV and RPD. Your calibration should reflect the full range of real-world samples your operation sees.

Conclusion: Understanding and Managing NIR Outliers

An outlier isn't automatically noise. Sometimes it's the most important data point in your set — the one that tells you a supplier is drifting, a dryer is failing, or an ingredient doesn't match its certificate. Your job isn't to delete the uncomfortable numbers. It's to understand them. Remove what you can prove is a measurement artifact, keep what reflects reality, and document both decisions so your calibration and your auditors both have a clear record. That's how your lab stays accurate under the conditions that matter most.

For more on building calibration models that hold up in food and feed production and understanding calibration metrics like RMSEP and RPD, those resources go deeper on the numbers behind the decisions covered here.

Free tool — Calibration Metrics Calculator: Enter your reference values and NIR predictions in the Calibration Metrics Calculator to compute RMSEP, RPD, R², and bias the way our course teaches it — with interpretation thresholds for grain, dairy, and feed. Open the Metrics Calculator →

Free tool — Model Diagnostics Calculator: Drop your spectra and predictions into the Model Diagnostics Calculator to flag outliers via Mahalanobis distance, leverage, and Q-residuals — the same diagnostics we walk through in Lesson 25. Open the Diagnostics Calculator →

NIR Troubleshooting Guide

SpectroScience students get access to the NIR Troubleshooting Guide — systematic approach to diagnosing poor predictions, instrument drift, and calibration failures. Available as a free download in the student resource library.

Access the PDF library

NIR Fundamentals Course — Lesson 31: Troubleshooting & Problem Solving

This lesson focuses on troubleshooting and problem-solving in NIR analysis, providing practical strategies for identifying and addressing issues that lead to outliers. Understanding these techniques can help professionals determine whether to accept or reject data based on the underlying causes of measurement discrepancies.

Explore Lesson 31 in the NIR Fundamentals course

Continue learning: NIR Spectroscopy Training Online | NIR Fundamentals Course — 32 Lessons