New: the AI firewall watches the whole platform and explains each incident in plain words. See how it works

Detecting outliers in sensor data without deleting the early warning

Deleting outliers can delete the failure you were watching for. Flag suspect readings, sort sensor faults from real drift, and filter only where the data is used.

To detect outliers in sensor data without losing early warnings, flag suspect readings instead of deleting them, then sort each one into a sensor fault or a process change. Sensor faults look like flatlines, single-reading spikes and impossible values. Real equipment trouble tends to build as a drift that persists and shows in a related measurement too.

Much outlier advice comes from data cleaning, where the goal is a tidy dataset. In maintenance, the reading that does not fit is often the one you installed the sensor for, so my first rule is that raw readings stay stored and filtering happens where the data is used.

How do you remove spikes from sensor readings without losing data?

Filter the view, not the store. Keep every raw reading, flag suspect ones, and apply the filter in the chart, report or alarm that reads them.

A moving average over five readings spreads a single spike of +50 into five outputs that are each 10 too high, so the spike becomes a small hill that looks like a real change. A median over the same five readings ignores up to two outliers in the window and still follows a real step once three of the five readings show the new value.

The alarm side is covered in keeping one bad reading out of your alarms, and the chart side in what an hourly average hides.

How do you tell a faulty sensor from a real change?

Look at the shape of the anomaly and check one other measurement. Sensor faults have shapes of their own, and a real process change often shows in more than one signal.

PatternLikely causeWhat to check
Flatline: the same value for longer than the process allowsStuck sensor, or a gateway repeating the last value after losing the deviceReading timestamps and the device’s own status
Lone spike that is gone at the next readingElectrical noise or a corrupted messageWhether neighbouring sensors moved at the same moment
Impossible value: outside the sensor’s range or the physicsWiring fault, open loop or failed sensorThe sensor’s datasheet range and the cabling
Step at a known timeSensor replaced or recalibratedThe maintenance log for that date
Slow drift that persistsReal wear or fouling, or sensor driftA related measurement, or a second instrument

The last row is where the money is. A bearing temperature creeping up while vibration rises on the same machine points to wear. The same creep with flat vibration and a sensor that has not been checked in years may be the sensor. I went through calibration for one sensor type in what soil moisture readings mean.

Anomaly detection for equipment maintenance: where should you start?

Start with rules you can explain, and add statistics later. A rule like “flatline for 30 minutes” or “value outside the sensor range” catches the common sensor faults and tells the technician what to look at.

  1. Set physical limits per sensor: its range, and the fastest rate of change the process can produce.
  2. Add a flatline check with a time limit that suits the signal’s normal noise.
  3. Compare each machine with its own history at a similar load, not with a fleet average.
  4. Where you have identical machines, compare them with each other.
  5. Only then try statistical scores, and send what they flag to a person to review rather than to the bin.

A statistical test flags whatever does not fit its idea of normal, and a developing fault is one of those things, so investigate what it flags. A test that learns “normal” from recent data can also absorb a slow drift into its baseline, which is why the rule-based checks stay in place. None of this replaces the machine’s own protection: trips and interlocks stay in the control system. For a worked set of condition points, see conveyor, crusher and ventilation fan monitoring in mining and the manufacturing industry page.

Frequently asked questions

Should outliers be excluded from report averages?

Exclude readings flagged as sensor faults from the calculation, keep them stored, and state in the report how many were left out. Readings that are only unusual, with no fault behind them, belong in the average, because they describe what the process did.

Can a z-score find faulty sensors?

Only some of them. A z-score measures how many standard deviations a reading lies from the mean, so it flags rare values, not broken sensors. A sensor stuck at a normal value has a z-score near zero and passes the test every time. Pair z-scores with a flatline check and range limits.

How many readings should a median filter use?

Use the smallest odd window that covers the longest spike you want to ignore. A window of three ignores one bad reading, a window of five ignores two in a row. Larger windows delay real changes by about half the window, so keep them short on signals where speed matters.

Share

See your field data live this week

Start a trial in your region with your first devices, or book a demo and bring your own use case.