How to Calculate RMSEP, SEP, and Bias from a Validation Dataset — Step-by-Step
RMSEP, SEP, and bias are the three numbers that determine whether your NIR calibration is ready for production. This article walks through the calculations from
How to Calculate RMSEP, SEP, and Bias from a Validation Dataset — Step-by-Step
If you are building a near infrared calibration for moisture in wheat, protein in soybeans, or fat in dairy powder, the moment of truth comes when you test that calibration on data it has never seen. That is what validation is for. You hold out a set of samples, run your model, and compare predicted values against the reference lab values. The numbers you get from that comparison — RMSEP, SEP, and bias — tell you whether your model is ready for the real world or whether it needs more work.
Many analysts know the acronyms but hesitate when it comes to the actual arithmetic. That is understandable, because the formulas look similar and the terminology shifts depending on which software you use. This guide walks you through how to calculate RMSEP, SEP, and bias from a validation dataset, step by step, with a worked example from a common food application. By the end, you will be able to compute these statistics by hand, understand what each one means for your calibration, and report them with confidence.
Why Validation Statistics Matter in NIR Spectroscopy
In near infrared spectroscopy, you build calibrations using a set of samples with known reference values. But a calibration that works on its training set can still fail on new samples. That is why you split your data — or collect a separate set — for validation. The validation dataset is your honest test. It tells you how well your model generalizes to samples it has not seen before.
The three statistics you will calculate from that validation set serve different purposes:
- RMSEP (Root Mean Square Error of Prediction) gives you the overall prediction error, combining both random error and any systematic offset.
- SEP (Standard Error of Prediction) isolates the random error after removing systematic bias.
- Bias tells you whether your model consistently over- or under-predicts.
Together, they give you a complete picture of model performance. A low RMSEP with a high bias means your model is precise but systematically off. A low bias with a high RMSEP means your model is centered but noisy. You need all three numbers to diagnose problems and decide whether to adjust your calibration or start over.
What You Need Before You Start
To calculate these statistics, you need two columns of data from your validation set:
- Reference values — the lab-measured values (e.g., moisture percentage by oven drying)
- Predicted values — the NIR model’s output for the same samples
Make sure your validation samples are truly independent. If you used the same samples for calibration and validation, your numbers will be artificially optimistic and you will get a false sense of security. A proper validation set should represent the range of variability you expect in routine use — different batches, different seasons, different temperatures if that matters for your product.
Use at least 20 representative independent validation samples as a minimum under ISO 12099; more may be needed depending on calibration complexity, analyte range, and validation objectives. With fewer samples, a single outlier can skew your results dramatically.
Step 1: Calculate the Prediction Errors
The first step is the same for all three statistics. For each sample in your validation set, subtract the predicted value from the reference value. This gives you the prediction error (also called the residual).
Formula:error_i = reference_i - predicted_i
Write these errors in a column. They will be positive when your model under-predicts and negative when it over-predicts.
Let us use a practical example. Suppose you are validating a calibration for moisture in corn. You have 10 validation samples with the following reference and predicted values:
| Sample | Reference (%) | Predicted (%) | Error |
|---|---|---|---|
| 1 | 14.2 | 14.5 | -0.3 |
| 2 | 15.1 | 14.9 | +0.2 |
| 3 | 13.8 | 14.0 | -0.2 |
| 4 | 16.0 | 15.6 | +0.4 |
| 5 | 14.5 | 14.7 | -0.2 |
| 6 | 15.3 | 15.1 | +0.2 |
| 7 | 13.5 | 13.8 | -0.3 |
| 8 | 14.8 | 14.6 | +0.2 |
| 9 | 15.6 | 15.2 | +0.4 |
| 10 | 14.0 | 14.3 | -0.3 |
Now you have everything you need to compute all three statistics.
Step 2: Calculate the Bias
Bias is the simplest statistic. It is simply the average of all your prediction errors. A positive bias means your model tends to under-predict (reference values are higher than predicted). A negative bias means your model tends to over-predict.
Formula:Bias = (sum of all errors) / n
where n is the number of validation samples.
Using our example:
Sum of errors = (-0.3) + 0.2 + (-0.2) + 0.4 + (-0.2) + 0.2 + (-0.3) + 0.2 + 0.4 + (-0.3)
Sum of errors = 0.1
Bias = 0.1 / 10 = 0.01% moisture
This is a very small bias — essentially zero, which is what you want. Your model is not systematically over- or under-predicting.
Step 3: Calculate SEP (Standard Error of Prediction)
SEP measures the random variation in your predictions after removing the bias. Think of it as the standard deviation of your prediction errors, but centered around the bias rather than around zero. This is an important distinction — if you have a large bias, SEP will be smaller than RMSEP because it removes that systematic offset.
Formula:SEP = sqrt( sum((error_i - bias)^2) / (n - 1) )
Note the use of (n - 1) in the denominator. This is the same correction you use when calculating a sample standard deviation. It accounts for the fact that you are estimating the bias from the same data.
Using our example:
First, subtract the bias from each error:
| Error | error - bias | (error - bias)^2 |
|---|---|---|
| -0.3 | -0.31 | 0.0961 |
| 0.2 | 0.19 | 0.0361 |
| -0.2 | -0.21 | 0.0441 |
| 0.4 | 0.39 | 0.1521 |
| -0.2 | -0.21 | 0.0441 |
| 0.2 | 0.19 | 0.0361 |
| -0.3 | -0.31 | 0.0961 |
| 0.2 | 0.19 | 0.0361 |
| 0.4 | 0.39 | 0.1521 |
| -0.3 | -0.31 | 0.0961 |
Sum of squared deviations = 0.7890
Divide by (n - 1) = 9: 0.7890 / 9 = 0.08767
Take the square root: SEP = sqrt(0.08767) = 0.296% moisture
So your SEP is about 0.30% moisture. This means that after removing the (tiny) bias, your model’s random prediction error has a standard deviation of about 0.3 percentage points.
Step 4: Calculate RMSEP (Root Mean Square Error of Prediction)
RMSEP is the most commonly reported validation statistic because it captures the total error — both random and systematic. It is calculated by squaring each error, averaging those squares, and taking the square root. The squaring gives more weight to large errors, which is useful because large errors are usually the ones that cause real problems in practice.
Formula:RMSEP = sqrt( sum(error_i^2) / n )
Note that RMSEP uses n in the denominator, not (n - 1). This is a key difference from SEP. RMSEP is a measure of the average magnitude of error, not an estimate of a population standard deviation.
Using our example:
Square each error:
| Error | error^2 |
|---|---|
| -0.3 | 0.09 |
| 0.2 | 0.04 |
| -0.2 | 0.04 |
| 0.4 | 0.16 |
| -0.2 | 0.04 |
| 0.2 | 0.04 |
| -0.3 | 0.09 |
| 0.2 | 0.04 |
| 0.4 | 0.16 |
| -0.3 | 0.09 |
Sum of squared errors = 0.79
Divide by n = 10: 0.79 / 10 = 0.079
Take the square root: RMSEP = sqrt(0.079) = 0.281% moisture
Step 5: Understand the Relationship Between the Three
Now that you have all three numbers, you can see how they relate. There is a useful mathematical relationship:
RMSEP² = SEP² + Bias²
Let us check with our example:
SEP² = 0.296² = 0.0876
Bias² = 0.01² = 0.0001
Sum = 0.0877
RMSEP² = 0.281² = 0.0790
Wait — these do not match exactly. That is because we rounded intermediate values. If you carry full precision through the calculation, the relationship holds. The small discrepancy here is purely from rounding. In practice, this identity is useful because it tells you how much of your total error comes from bias versus random variation.
In our example, bias is essentially zero, so RMSEP and SEP are nearly identical. That is the ideal situation. But consider a case where bias is large — say 0.5% moisture. Then RMSEP would be noticeably larger than SEP, and you would know that your model has a systematic problem that needs correcting.
Practical Considerations for Food and Agriculture NIR
In real-world NIR work, you will rarely calculate these statistics by hand. Most software packages — whether from the instrument manufacturer or third-party chemometrics tools — report them automatically. But understanding the calculations helps you interpret the numbers critically.
For moisture in grains, an RMSEP below 0.25% is generally considered excellent. For protein in wheat, you might accept an RMSEP of 0.3–0.4%. For fat in dairy powder, the acceptable range depends on the specification limits you need to meet. A good rule of thumb is that your RMSEP should be no more than one-third to one-half of the tolerance you need to enforce. If you need to guarantee moisture below 15% and your RMSEP is 0.5%, you will have trouble making confident decisions near the limit.
Bias is often correctable. If your validation shows a consistent bias, you can apply a simple bias correction to your model — subtract the bias from all future predictions. This is a standard practice in many labs. However, if bias appears only in validation but not in calibration, it may indicate that your validation samples are systematically different from your calibration samples (e.g., different harvest season, different variety). In that case, you may need to broaden your calibration set rather than just applying a correction.
SEP tells you about precision. If
Continue learning: NIR Spectroscopy Training Online | NIR Fundamentals Course — 32 Lessons