How to Run a Lab Error Study for NIR: Calculating SEL Step-by-Step

Why Your NIR Calibration Is Only as Good as Your Lab

If you’ve spent any time with near-infrared spectroscopy, you’ve heard the phrase “garbage in, garbage out.” But what does that really mean in practice? It means your NIR instrument can only be as accurate as the reference method you use to build and validate its calibrations. If your lab reference values are noisy, your NIR predictions will be noisy too, no matter how sophisticated your chemometric model is.

That’s where the Standard Error of the Laboratory, or SEL, comes in. SEL quantifies the random variation inherent in your reference lab method — the unavoidable noise from sample prep, operator technique, instrument drift, and environmental conditions. Without knowing your SEL, you’re essentially flying blind when you assess NIR accuracy. You might chase a bias that isn’t real, or accept a calibration that’s worse than your lab noise allows.

In this guide, I’ll walk you through how to run a lab error study for NIR, step by step. You’ll learn how to calculate SEL, what the numbers mean, and how to use them to set realistic performance targets for your NIR instruments. We’ll use a practical example from grain and flour analysis, a common application in food and agriculture.

What Is SEL and Why Should You Care?

SEL stands for Standard Error of the Laboratory. It’s a statistical estimate of the repeatability and reproducibility of your reference method under routine conditions. Think of it as the baseline noise floor for your lab. If your lab can’t reproduce a protein measurement within ±0.1%, then your NIR can never claim to be better than that.

There are two related concepts you’ll see in NIR literature:

• SEL (Standard Error of the Laboratory): The random error associated with a single lab measurement, typically estimated from duplicate or replicate analyses.
• SELr (Repeatability) vs. SELR (Reproducibility): Repeatability is variation within a single lab, same operator, same instrument, short time span. Reproducibility is variation across labs, operators, and time. For most NIR calibration work, you’ll focus on SEL from duplicate measurements within your own lab.

The key insight is this: SEL sets the lower bound for your NIR standard error of calibration (SEC) and standard error of prediction (SEP). If your SEC is much smaller than your SEL, you’ve likely overfit the model. If your SEP is much larger than your SEL, you have room to improve your NIR method or your calibration samples.

Planning Your Lab Error Study

Before you start running samples, you need a clear protocol. A sloppy study gives you a misleading SEL, and that’s worse than no study at all. Here’s what to plan for.

Sample Selection: Represent Your Range

Your study needs to cover the same range of analyte concentrations you expect in routine use. Don’t just use 10 samples clustered around the mean. You want samples spanning the low, middle, and high ends of your calibration range.

For our flour example, that means collecting samples with protein content from, say, 8% to 16% (as-is basis). Aim for at least 20 to 30 unique samples. More is better, but 20 is a workable minimum.

Replication Design: The Duplicate Method

The simplest and most common design is duplicate measurements. You analyze each sample twice, under routine conditions, on the same day or across two days. This gives you a direct estimate of repeatability.

A more robust design uses triplicates or even a balanced incomplete block design, but for most food and ag labs, duplicates are sufficient. The math gets more complicated with more reps, and the gain in precision is marginal once you have 20+ samples.

Operator and Environmental Conditions

Here’s a subtle but critical point: your SEL should reflect routine conditions, not ideal conditions. If you run your study with one highly experienced operator on a perfectly calibrated instrument in a climate-controlled room, your SEL will be artificially low. Then, when your NIR performs worse than expected in the real world, you won’t know why.

Instead, have two or three operators run the samples, ideally on different days. Let them follow their normal procedures. Don’t give them any special instructions beyond what they’d do on a regular shift. This gives you a realistic SEL that accounts for normal human and environmental variation.

Step-by-Step: Running the Study

Let’s walk through the actual procedure. I’ll use a flour protein example, but the steps apply to any food or agricultural matrix — meat, dairy, grains, oilseeds, feed, you name it.

Step 1: Collect and Prepare Your Samples

Gather your 20–30 unique flour samples. Make sure they’re properly labeled and stored to prevent moisture changes. Grind or homogenize as you would for routine analysis. If your NIR method uses whole grain, use whole grain for the lab reference too — don’t mix sample prep protocols.

Step 2: Assign Random Order

Randomize the order of analysis. This prevents any systematic drift (e.g., instrument warm-up, reagent degradation) from biasing your results. You can simply shuffle the sample IDs and run them in that shuffled order.

Step 3: Run the First Replicate

For each sample, run your reference method (e.g., Kjeldahl or Dumas for protein) and record the result. Follow your standard lab protocol exactly. Don’t rush, don’t take shortcuts, but also don’t do anything extra special. This is a routine measurement.

Step 4: Run the Second Replicate

After you’ve run all samples once, run them again in a different random order. Ideally, do this on a different day. If you’re using the same day, at least wait a few hours and re-randomize. This second pass captures day-to-day or session-to-session variation.

Step 5: Record Your Data

Create a simple spreadsheet with three columns: Sample ID, Replicate 1, Replicate 2. Here’s a small example to illustrate:

Sample ID	Protein Rep 1 (%)	Protein Rep 2 (%)
F-01	10.52	10.61
F-02	12.18	12.09
F-03	9.87	9.95
F-04	14.33	14.21
F-05	11.04	11.12
…	…	…

Calculating SEL: The Math

Now for the part you’ve been waiting for. The formula for SEL from duplicate measurements is refreshingly simple:

SEL = √( Σ(d²) / (2n) )

Where:

• d = the difference between the two replicates for each sample (Rep 1 – Rep 2)
• d² = that difference squared
• Σ = sum of all squared differences
• n = the number of samples (pairs)

Let’s work through the example above with 5 samples (in practice you’d use 20–30, but this keeps the math clear).

First, calculate the differences and squared differences:

Sample	Rep 1	Rep 2	d = Rep1 – Rep2	d²
F-01	10.52	10.61	-0.09	0.0081
F-02	12.18	12.09	0.09	0.0081
F-03	9.87	9.95	-0.08	0.0064
F-04	14.33	14.21	0.12	0.0144
F-05	11.04	11.12	-0.08	0.0064

Now sum the squared differences:

Σ(d²) = 0.0081 + 0.0081 + 0.0064 + 0.0144 + 0.0064 = 0.0434

With n = 5:

SEL = √( 0.0434 / (2 × 5) ) = √( 0.0434 / 10 ) = √(0.00434) = 0.0659%

So in this small example, the SEL for your lab’s protein method is about 0.066%. That means the typical random error in a single lab measurement is roughly ±0.07% protein.

Interpreting Your SEL Value

Now that you have a number, what do you do with it? Here’s where the real value of the study shows up.

Setting Realistic NIR Performance Targets

Your NIR calibration’s standard error of prediction (SEP) should be no better than your SEL. In fact, a good rule of thumb is that SEP should be roughly 1.0 to 1.5 times your SEL. If your SEP is much larger, your NIR is adding error beyond the lab noise — you have a calibration problem. If your SEP is much smaller than your SEL, that’s a red flag. It suggests your calibration is overfit or your validation samples aren’t truly independent.

In our flour example, with an SEL of 0.066%, you’d want your NIR protein calibration to have a SEP somewhere in the range of 0.07% to 0.10%. That’s a realistic, defensible target.

Comparing Instruments and Methods

SEL is also useful for comparing different NIR instruments or different reference methods. If you’re deciding between two NIR units, run the same samples on both and compare their SEPs against your SEL. The instrument that gets closer to the SEL floor is the better performer.

Similarly, if you’re considering switching reference methods (e.g., from Kjeldahl to Dumas), run a quick SEL study on both. The method with the lower SEL will give you a cleaner target for NIR calibration.

Troubleshooting Poor NIR Performance

If your NIR predictions are worse than expected, your first question should be: “Is the problem in the NIR or in the lab?” Run a quick SEL study on your reference method. If the SEL is high (say, 0.15% protein), then your NIR might actually be performing near the theoretical limit — the lab noise is the bottleneck, not the NIR.

This is a common scenario in busy production labs. Operators are rushed, sample prep varies, and the reference method drifts. A lab error study often reveals that the “poor” NIR performance is actually excellent, given the noisy reference data.

Common Pitfalls to Avoid

Let me save you some headaches. Here are the mistakes I see most often in lab error studies.

Using Too Few Samples

Ten samples or fewer gives you a very unstable estimate of SEL. The confidence interval around your SEL will be wide, and you might make bad decisions based on a number that’s not reliable. Stick with at least 20, ideally 30.

Pooling Replicates Incorrectly

Some people make the mistake of running all replicates in a single batch, back-to-back. That measures instrument repeatability, not method repeatability. You want to capture the full routine process, including sample prep, weighing, digestion, and reading. Spread your replicates across days and operators.

Ignoring Outliers

If you see a huge difference between two replicates for one sample, don’t just delete it. Investigate. Was there a labeling error? Did the sample sit out too long? Is there a genuine heterogeneity issue with that sample? If you can identify a clear procedural error, you can exclude it, but document it. If you can’t explain it, keep it in — that’s real-world noise.

Confusing SEL with Standard Deviation

SEL is not the same as

Continue learning: NIR Spectroscopy Training Online | NIR Fundamentals Course — 32 Lessons