Outlier

A outlier, o outlier, is a data point or set of data points that deviates significantly from the rest of the observations in a dataset. In other words, if you’re wondering what is atypical, refers to those values that do not follow the general pattern of the rest.

The translation of *Outlier* In Spanish, this is «valor atípico», and understanding this translation correctly is key to interpreting statistical analyses accurately.

In statistics, the outliers They may arise due to natural variability in the data, measurement errors or data entry errors, or they may represent rare but significant events. Meaning of ‘outliers’ This implies that these values may be either an indication of something of interest or simply an error, depending on the context. Identifying and handling outliers is essential, as they can have a significant impact on the results of statistical analyses and predictive models.

When do we consider a data point to be an outlier?

There are several methods for identifying outliers:

• Interquartile range (IQR) method: A piece of data is considered a outlier if it lies more than 1.5 times the interquartile range above the third quartile or below the first quartile. This method is useful for data that do not follow a normal distribution.
• Distance from the mean: For data that follow a normal distribution, a data point may be regarded as a outlier if it lies more than 2 or 3 standard deviations from the mean.
• Scatter plots or box plots: Visually, the outliers They can be identified as points that lie far from the rest of the data on a scatter plot or that fall outside the «whiskers» of a box plot.

Problems with outliers in machine learning

In the context of machine learning, the outliers can pose a significant challenge for a number of reasons:

• Model distortion: Machine learning models, particularly those based on distance such as k-NN or regression models, can be very sensitive to outliers. These outliers may bias the model, affecting its accuracy and generalisability.
• Noise and errors: The outliers These can often be noise or errors, which can cause the model to learn irrelevant or incorrect patterns. This can reduce the quality of the model and lead to overfitting.
• Impact on model evaluation: In classification tasks, the outliers They may affect evaluation metrics such as precision, recall and F1-score, as they may be misclassified more frequently.

To mitigate these problems, techniques such as data normalisation or standardisation, and the removal of outliers extreme values, or the use of robust algorithms that are less sensitive to these values.

Let us consider a dataset of salaries within a company. If the majority of employees earn between 30,000 and 50,000 dollars a year, a CEO earning 500,000 dollars a year would be a clear outlier.

This figure is not representative of the typical salary at the company and could skew any analysis or model that attempts to predict salaries based on this data.

The outliers are an important part of the data analysis and machine learning. Whilst they can provide valuable information, they also have the potential to distort results and lead to incorrect conclusions if not handled properly.

Identify, analyse and, if necessary, treat the outliers It is a key step in obtaining accurate and reliable insights from any data analysis.