In statistics, researchers draw a sample from a population and use their observations to make generalizations about the entire population. For example, you might present a subset of visitors with different versions of a web page to get an estimate of how all visitors to the page would react to each version.
Because there’s always random variability, or error, the sample can’t be expected to be a perfect representation of the population. However, if it's a reasonably large, well-selected sample, you can expect that the statistics you calculate from it are fair estimates of the population parameters.
Even so, certain factors can influence the sampling and collection of data, causing the resulting statistic or model to be unrepresentative of the population. These factors, or biases, are common and can result in unreliable analyses.
What Is Statistical Bias?
Statistical bias is anything that leads to a systematic difference between the true parameters of a population and the statistics used to estimate those parameters. In other words, bias refers to a flaw in the experiment design or data collection process, which generates results that don’t accurately represent the population.
In business, statistics are commonly used to aid the decision-making process. For example, a manager at a healthcare clinic might use historical data to project how many patients are expected to visit in a week to estimate staffing needs. Several forms of bias, however, have the potential to impact this analysis, causing the manager to make a decision based on faulty information—and possibly bring serious consequences to the business.
Here are three of the most common types of bias and what can be done to minimize their effects.
Related: The Advantages of Data-Driven Decision Making
Types of Statistical Bias to Avoid
1. Sampling Bias
In an unbiased random sample, every case in the population should have an equal likelihood of being part of the sample. However, most data selection methods are not truly random.
Take exit polling, for example. In exit polling, volunteers stop people as they leave a polling place and ask them who they voted for. This method leads to the exclusion of those who vote by absentee ballot. Furthermore, research suggests the volunteers are more likely to gather data from people similar to themselves.
Polling volunteers are more likely to be young, college-educated, and white compared to the general population. It's understandable that a white college student would be more likely to approach someone who looks like they could be one of their classmates than a middle-aged woman, struggling to keep three children under control. This means not every person has the same chance of being selected for an exit poll.
2. Bias in Assignment
In a well-designed experiment, where two or more groups are treated differently and then compared, it’s important that there aren’t pre-existing differences between groups. Every case in the sample should have an equal likelihood of being assigned to each experimental condition.
Let's say the creators of an online business course think that the more times they get a visitor to come to their website, the more likely they are to enroll. In fact, people who visit the site five times are more likely to enroll than people who visit three times, who are, in turn, more likely to enroll than people who visit only once.
One might mistakenly conclude that more site visits lead to more enrollment. However, there are systematic differences between the groups that precede their visits to the site. The same factors that motivate a potential student to visit the site five times, rather than once, may also make them more likely to enroll in the course. Because each person didn’t have an equal chance of being in each experimental group—visiting the site five, three, and one time, respectively—it can’t be concluded that the number of site visits leads to enrollment in the course.
3. Omitted Variables
When analyzing trends in data, it’s important to consider all variables, including those not accounted for in the experimental design. Just because two variables are correlated doesn’t mean one caused the other—there could be additional variables at play.
For example, in 1980, Robert Matthews discovered an extremely high correlation between the number of storks in various European countries and the human birth rates in those countries. Using Holland as an example, where only four pairs of storks were living in 1980, the birth rate was less than 200,000 per year; while Turkey, with a shocking 25,000 pairs of storks had a birth rate of 1.5 million per year.
The correlation between the two variables was an extremely significant 0.62. Matthews used this example—drawing from the myth that storks deliver newborn babies—to illustrate that correlation doesn’t imply causation. The high correlation between the two variables doesn’t imply that a high stork population causes an increase in birth rate. Rather, there’s a third variable at play: geographic area. Large countries have more people living in them—hence higher birth rates and a higher stork population.
Rerunning the analysis and including area as an independent variable solves this mystery. While it may not be possible to identify all omitted variables, a good research model explores all variables that might impact the dependent variable.
4. Self-Serving Bias
One phenomenon to keep in mind when analyzing survey data is self-serving bias. When asked to self-report, people tend to downplay the qualities they perceive to be less desirable and overemphasize qualities they perceive to be desirable.
For example, a study might find a strong positive correlation between being a good driver and being good at math. However, if the data were collected using a self-report tool, such as a survey, the correlation could be a side effect of self-serving bias. People who are trying to present themselves in the best possible light might overstate their driving ability and their math aptitude.
5. Experimenter Expectations
If researchers have pre-existing ideas about the results of a study, they can accidentally have an impact on the data, even if they're trying to remain objective. For example, interviewers or focus group facilitators can subtly influence participants through unconscious verbal or non-verbal cues.
Experimenter effects have even been observed with non-human participants. In 1907, a horse named Clever Hans was famous for successfully completing complex mathematical operations and tapping out the answer with his hoof. It was later discovered that he was responding to the involuntary body language of the person posing the problems. To avoid experimenter bias, studies that require human intervention to gather data often use blind data collectors who don't know what’s being tested.
Better Data for Better Business Decisions
Although it’s difficult to completely avoid bias, it’s critical that analysts, data scientists, and other business professionals are aware of its sources so they can minimize its effects.
Paying close attention to the data collection process and analysis can help you identify possible flaws and reduce their impact on the final results. Doing so can lead to better models and more reliable insights. Armed with these insights, you can make data-backed business decisions that keep your organization moving in the right direction.
To learn how to unlock the power of your organization’s data, explore our eight-week online course Business Analytics or download our Beginner’s Guide to Data & Analytics.
This post was updated on February 2, 2021. It was originally published on June 13, 2017.
