Artificial intelligence shapes decisions in healthcare, hiring, education and public services. What happens when the systems behind those decisions carry hidden biases?
It starts with the data
Take a familiar example. Estonian has no grammatical gender. Translate “someone is an engineer and someone is a teacher” into English, and the result has often been “he is an engineer” and “she is a teacher.” This is not malicious. It reflects what the training data contains. That data reflects decades of societal patterns.
This kind of historical bias is just the starting point. In their 2021 framework, researchers Suresh and Guttag identified eight types of bias that can enter an AI system. They grouped these by stage. During data generation alone, three types can appear: historical bias baked into the world itself, sample bias from who ends up in the dataset, and measurement bias from how data is collected and labelled.
Vulnerable groups (the elderly, ethnic minorities, lower-income communities) are often underrepresented in training data. “Data altruists” who donate data for research often come from a narrow demographic. This skews things in the other direction.
The problem scales with AI
Statisticians have long understood the importance of representative samples. But AI systems now operate in schools, hospitals and hiring processes. The people building these tools do not always have the same statistical training. The stakes are higher. The scale is bigger. The safeguards have not kept pace.
A clear example comes from medicine. Drug trials have long skewed towards male participants. Men’s stable hormone cycles make them easier to study. The result is that medications can work differently, or less well, for women. When AI systems learn from this imbalanced data, they repeat the same gaps at greater scale.
Beyond the data: technical and deployment bias
Even with perfect data, bias can enter during model training, evaluation and deployment. The wrong algorithm for a task, or a model tested against an unsuitable benchmark, introduces what researchers call learning bias and evaluation bias.
Deployment bias is a growing concern. A model fine-tuned on US consumer behaviour gets used for educational recommendations in India. Chatbots trained in one cultural context transmit that culture’s values worldwide. Children now turn to AI with questions they once asked their parents, from career advice to everyday life decisions. The cultural assumptions in these systems influence people at a personal level.
The Dutch SyRI case: a cautionary tale
One of the most cited examples of AI bias in practice is SyRI (System Risk Indication). The Dutch government used this system from 2008 to 2020 to detect social benefit fraud. SyRI was a rule-based decision tree, not a machine learning model. It still disproportionately flagged people from vulnerable groups, including ethnic minorities and low-income communities. Some were falsely accused of fraud. They lost their benefits. Some had children taken into care. In some cases, people took their own lives.
A European court struck down the system. What makes this case instructive is that the bias was not hidden in opaque algorithms. People designed it into the rules. Questions like “do you live above a bar?” or “is your car older than seven years?” carried built-in class and demographic biases.
Human oversight is not a silver bullet
“Human in the loop” is often presented as the answer to AI bias. But when a system is correct 99% of the time, the human reviewer grows complacent. They rubber-stamp decisions rather than question them.
In the SyRI case, a communication gap between teams meant each assumed the other had verified the results. Human oversight works only when people understand their role and stay engaged. It fails when treated as a checkbox exercise.
Design bias: the one that is always there
Design bias deserves the most attention of all. System creators bring their own biases simply by being human. Even with perfect data, perfect training and perfect deployment, the people who design the system carry their own assumptions, blind spots and cultural views. Everyone has biases, shaped by upbringing, education and lived experience. No single universal system of ethics exists.
This means bias can enter a system even when nobody intended it to.
What organisations can do
The most practical way to tackle AI bias is a structured risk management framework: context establishment, risk assessment and risk treatment, adapted for bias.
As part of the Equititech project, researchers at Cybernetica developed a methodology and guideline for AI bias risk management. They built this for the Estonian Ministry of Justice and Digital Affairs. The key resource is a workbook (now in spreadsheet form, with a web-based version planned). It provides prompting questions and examples to help teams identify and evaluate bias risks in their systems.
The recommendation is to assemble a diverse group with legal, technical, societal and managerial views. They work through the assessment together. No dedicated bias expert is needed. The diversity of viewpoints in the room is what matters.
Not all bias calls for the same response. Risk scoring helps organisations set priorities: what is the likelihood of a biased outcome, and what is the impact if it occurs? Some risks demand immediate action. Others can be addressed over time. In some cases, the right answer is to stop using the system.
Key takeaways
- AI bias is not just a data problem. It can enter at every stage from data collection to deployment, and through system design choices made by humans.
- Historical and societal inequalities grow when AI systems learn from biased data and deploy at scale.
- Human oversight is valuable but not enough on its own. Reviewers need clear responsibilities and must stay actively engaged.
- A structured risk management approach is the most practical way for organisations of any size to identify and address bias.
- Tools and guidelines are now available, including the Equititech project’s bias risk management workbook for Estonian public and private sector organisations.
Referenced research: Suresh, H., & Guttag, J. (2021). A framework for understanding sources of harm throughout the machine learning life cycle. This paper defines the eight types of bias discussed in this article.
Explore Cybernetica’s AI solutions
This article is based on an episode of Science will make you happy, a podcast series by Cybernetica exploring how science and technology shape the world around us.