• AI bias refers to systematic error or prejudice in AI outputs that unfairly favours or disadvantages certain groups or individuals 🔗
  • Bias can stem from human biases transferred to AI (through data or design) or from technical flaws.
• Fairness, conversely, means outcomes that are impartial and justifiable across different groups.
  • Achieving fairness often requires identifying and managing biases.

• AI systems are increasingly used to make decisions that affect people’s lives
  • who gets 👩🏾‍🔧 hired, approved for a 🏦 loan, admitted to the 🏥 ICU, or monitored by 🕵🏽 police.
• Unfair biases can lead to discrimination, harming individuals and society

Adapted from AI Multiple Research:

Beyond the (arguably more important) moral and ethical costs, there are financial costs as well 🔗.

• Regulatory pressure: New regulations hold companies accountable for AI bias.
  • E.g., New York City now requires annual bias audits for AI hiring tools 🔗
• Public expectations: Clients and employees demand fair, responsible AI.
  • Organizations championing fairness can differentiate themselves and avoid scandals.
• Long-term success: Fair AI leads to better decisions and broader market reach.

tl;dr: ethical AI is good for business.

• An image-to-text neural network produces a novel, readable, coherent report given new image data (Liu et al. 2019).
  • It combines a convolutional neural network (CNN) for image understanding with a recurrent neural network (RNN) for understanding sequences (of text)

• Higher model underdiagnosis rates on one subpopulation, such as female patients, would lead to a higher rate of no treatment for those patients if the model were deployed.
• Seyyed-Kalantari et al. (2021) studied whether “AI systems may reflect and amplify human bias”

Largest underdiagnosis rates in Female, 0-20, Black, and Medicaid insurance patients.

• Bias in AI doesn’t come from nowhere — it’s a mirror of our data, decisions, and designs.
• Understanding where bias originates is the first step to preventing harm.
  • Bias can creep in at multiple points in the AI lifecycle: from the moment we collect data to when we deploy models in the real world.
• We’ll explore 5 key sources of bias:
  1. Data Bias
  2. Algorithmic Bias
  3. Human & Cognitive Bias
  4. Deployment & Context Bias
  5. Feedback Loops & Systemic Bias

Let’s unpack each one — with real-world examples and academic grounding.

• AI is only as good as the data it learns from. But real-world data is rarely neutral.
• Common types of data bias include:
  • Historical bias: Data reflects past inequalities (e.g., biased hiring practices).
    • e.g., hiring records from 1960s-1980s dominated by men leads AI to favour male candidates.
      
  • Sampling bias: Some groups are over- or under-represented.
    • e.g., data on skin lesions sampled from University-educated participants only
  • Label bias: Subjective or inaccurate labels
    • e.g., police records reflect policing patterns, not crime patterns.

📚 Recent literature: Barocas, Hardt, and Narayanan (2023)

• Even with perfect data, algorithms can introduce bias:
  • Objective function bias: Optimizing for accuracy may ignore fairness (e.g., one group may have higher false positives).
  • Feature selection bias: Algorithms may rely on proxies for sensitive attributes (e.g., postal codes as stand-ins for race).
  • Model complexity: Black-box models may hide biased correlations that are hard to detect and correct.

• Behind every AI system (for now) is a team of humans — with all our assumptions, shortcuts, and blind spots.
  • Confirmation bias: Interpreting results to match expectations.
  • Implicit bias: Unconscious preferences embedded in decisions (e.g., labeling, feature design).
  • Organizational bias: Incentives that prioritize speed, profit, or efficiency.

• Even a well-trained model can behave unfairly in the wild.
  • Domain shift: Real-world users differ from training data (e.g., accents, behaviour, devices).
  • Use mismatch: AI is applied in settings it wasn’t trained or tested for.
  • Automation bias: People over-trust AI decisions, even when they’re wrong.

• AI doesn’t just predict the world — it shapes it.
  • Reinforcement of bias: Biased predictions affect decisions, which produce data that reinforce the original bias.
  • Echo effects: Cesspools social media may push similar content based on prior engagement, narrowing exposure.
  • Institutional feedback: Predictive policing algorithms trained on past arrest data can lead to over-policing of certain communities — which in turn generates more arrest data from those same areas.
    • e.g., predictive policing tools like PredPol focused patrols on neighborhoods with high historical arrest rates, reinforcing biased enforcement patterns 🔗.

• Context:
  • Commercial algorithm used to identify high-risk patients for additional care.
  • Widely deployed in U.S. healthcare systems, affecting ~200 million people annually.
• Key Findings:
  • At the same risk score, Black patients were significantly sicker than White patients.
  • Black patients were less likely to be enrolled in care programs due to underestimated risk.
  • Algorithm was trained to predict future healthcare costs, not health status.
• Source of Bias:
  • Cost ≠ Health Need: Due to unequal access, Black patients often incur lower healthcare costs despite worse health.
  • Algorithm learned to associate lower costs with lower risk, systematically underestimating needs of Black patients.
• Impact:
  • Fixing the bias could raise Black patient enrollment from 17.7% to 46.5%.

• Context:
  • Study of ~7M US mortgage applications (2009–2015), in traditional and FinTech lenders.
  • Algorithms increasingly used in consumer credit decisions, often marketed as “bias-free”.
• Key Findings:
  • Minority borrowers (Black and Latinx) paid 7.9 bps more on purchase mortgages and 3.6 bps more on refinance mortgages—even after controlling for credit risk.
  • This pricing disparity translates to $765M/year in extra interest paid by minority borrowers.
  • FinTech lenders exhibited ~40% less discrimination in pricing, but did not eliminate it.
• Source of Bias:
  • Algorithms used non-credit-risk variables (e.g. shopping behaviour, location) to strategically price loans.
  • Such variables can proxy for race, causing a disparate impact—even when not explicitly racial.

• Impact:
  • Estimated 0.74M–1.3M minority applications were rejected due to bias during the 7-year period.
  • FinTech lenders showed no discrimination in loan approval decisions, suggesting potential for improvement.

📚 Reading: (Bartlett et al. 2019)

• In the mid-2010s, Amazon built an internal tool to automate résumé screening for engineering roles. It trained on a decade of résumés — most of which came from male applicants, reflecting historical gender imbalances in tech.
• The AI downgraded resumes that mentioned “women’s” (e.g., “women’s chess club” or “women’s college”).
  • It favoured resumes that resembled those of male applicants, even when irrelevant to job performance.
  • Amazon scrapped the project after realizing the model was replicating and reinforcing gender bias.
• 🔗Reuters

• Wider context: Many companies trialing AI for hiring have found biases:
  • Facial analysis in video interviews showing bias against certain ethnicities or accents.
  • Job ads targeting: On social media, ads for high-paying jobs were shown more to men than women until policies were changed
• Impact: Hiring biases mean qualified people from marginalized groups are overlooked, and companies miss out on talent. It also opens firms to discrimination claims.

• 📸 Facial Recognition (FR) is often presented as a cutting-edge tool for identity verification and law enforcement. But without regulation or robust validation, it can produce harmful, racially biased outcomes.
• 🔬 Accuracy Disparities: Landmark study Gender Shades (Buolamwini and Gebru 2018) found:
  • Error rates <1% for light-skinned men
  • ❗ Over 34% for dark-skinned women
  • 📚 See: 🔗 Gender Shades
• Poor performance stems from imbalanced training data — typically skewed toward lighter male faces.

• ⚠️ Real-world consequence:
  • In 2020, Robert Williams, an innocent Black man in Detroit, was wrongfully arrested due to FR error.
  • Police acknowledged the mistake and updated policy to require corroborating evidence beyond a facial match.
• 🧠 Policy Risks & Regulatory Gaps:
• Goodall (2016) warned that FR adoption outpaced policy development:
  • No standardized benchmarks for accuracy or fairness
  • Fragmented oversight across law enforcement and government sectors

• ⚖️ AI systems are increasingly used in government decision-making — from immigration triage to criminal sentencing.
  • 🇬🇧 The UK Home Office used an AI-assisted tool to “stream” visa applicants into risk categories
  • Advocacy groups found that the system relied on discriminatory country-based criteria, reinforcing a “hostile environment”
  • The algorithm wasn’t transparent, but critics dubbed it “speedy boarding for white people.”
  • After legal pressure, the Home Office abandoned the system in 2020 🔗
  • To be fair, more recent tools have shown no negative impact on decision quality 🔗

• ⚖️ Many U.S. jurisdictions use AI scoring systems like COMPAS to inform bail, sentencing, and parole decisions.
  • A landmark investigation by 🔗 ProPublica revealed that COMPAS was:
  • Twice as likely to falsely label Black defendants as “high risk” compared to white defendants
  • Less accurate in predicting future re-offending for Black individuals

• Things are not so bleak. There are solutions.
  • Some of these will be ‘top-down’ and procedural.
  • Some will be ‘bottom-up’ and technical.

This may be a false dicohtomy

Source: McKinsey & Company

• Bias audits & testing: Systematically evaluate AI on diverse test data:
  • Slice analysis: Check performance separately for various demographics (e.g., accuracy for each gender, each ethnicity).
  • Adversarial testing: Use specially crafted or synthetic inputs designed to probe biases (e.g., test a CV-screening AI with identical resumes except for names indicating different demographics).
  • Human-in-the-loop: Have domain experts and affected groups review AI decisions for fairness.
  • Regulatory trend: Bias audits are becoming standard. E.g., 🔗 New York City’s bias audit law requires external audits of hiring algorithms for disparate impact, and results must be public.

• To manage bias, first we must detect and measure it. Quantitative measures to assess how outcomes differ across groups. For example:
  • Demographic parity: Does the system give positive outcomes (e.g. loan approved) at equal rates for different groups?
  • Equalized odds: Are error rates (false positives/negatives) similar for each group?
  • Counterfactual fairness: Would a decision be the same if a person belonged to a different demographic group (all else equal)?
• In practice, organizations decide which fairness criteria matters for their context (sometimes there are trade-offs).

• Equalized odds allows classification to differ only through the ground truth.
  • I.e., a binary classifier \(\color{orange}{R}\) for a set of groups \(\color{green}{S}\) if, for ground truth \(\color{blue}{Y}\) and group membership \(\color{purple}{A}\):
    • \(P\left(\color{orange}{R}=+\,|\, \color{blue}{Y=y}, \color{purple}{A=a}\right) = P\left(\color{orange}{R}=+\,|\, \color{blue}{Y=y}, \color{purple}{A=a'}\right)\)
      • \(\forall \color{blue}{y} \in \{0, 1\}\ \&\ \forall \color{purple}{a\neq a'} \in \color{green}{S}\)
  • I.e., True Positive and True Negative rates should be equal across groups.

🎯 Suppose you’re building a loan approval model and want it to be fair across race. You test two metrics:

• Demographic Parity: \(P\left(\color{orange}{R}=+\,|\, \color{purple}{A = a}) = P(\color{orange}{R}=+\,|\, \color{purple}{A=a'}\right)\)
  • Meaning: both groups get approved at the same rate.
• EO: \(P\left(\color{orange}{R}=+\,|\, \color{blue}{Y=y}, \color{purple}{A=a}\right) = P\left(\color{orange}{R}=+\,|\, \color{blue}{Y=y}, \color{purple}{A=a'}\right)\)
  • Meaning: error rates (FP and FN) are equal across groups.

The tradeoff (from Kleinberg, Mullainathan, and Raghavan (2016)):

You cannot, in general, satisfy both metrics unless base rates are equal: \(P(\color{blue}{Y=1}\,|\, \color{purple}{A=a}) = P(\color{blue}{Y=1}\,|\, \color{purple}{A=a'})\)

• In other words:
  • If one group is less likely to repay loans (different base rate), EO could lead to different approval rates (violating Demographic Parity).
  • Enforcing Demographic Parity will force you to tolerate unequal errors, potentially approving riskier loans for one group.

• In Module 1, we talked about transformers.
• The simple language models on which they’re based provides some insight as to how language models can be biased (Bolukbasi et al. 2016)

Bolukbasi et al. (2016)

Bolukbasi et al. (2016)

Chen et al. (2020)

• A growing ecosystem of open-source toolkits supports bias detection and mitigation across the AI lifecycle.
• These tools offer metrics, visualizations, and correction techniques — but must be used with human oversight.

• IBM AI Fairness 360 (AIF360) 🔗 research.ibm.com
  • A comprehensive Python library with over 70 fairness metrics and mitigation algorithms.
  • Includes pre-, in-, and post-processing strategies like re-weighting, bias-aware learning, and output adjustments.
• What-If Tool 🔗 pair-code.github.io/what-if-tool
  • A visual interface for TensorBoard.
  • Allows you to simulate counterfactuals (e.g., “What if this applicant were male instead of female?”)
    
  • Supports slicing data by demographic group and observing decision boundaries.
    

• Fairlearn 🔗 fairlearn.org
  • Focuses on fairness in model selection and post-processing.
  • Lets you visualize trade-offs (e.g., accuracy vs. equal opportunity) and enforce fairness constraints.
• Aequitas 🔗 github.com/dssg/aequitas
  • Audit toolkit that calculates disparity metrics across groups and generates fairness reports.
    
  • Widely used by public policy and government agencies.
    

AIMultiple catalogs a few additional tools:

Each of these vendors provides a different layer of oversight, from dataset management to LLM monitoring and enterprise AI governance.

• Tools can detect disparities — but cannot determine what’s ethically or legally acceptable in a given context.
• Pair tool outputs with:
  • Model cards (performance documentation across subgroups)
  • Organizational review boards

💡 Reminder: Fairness is not “one metric to rule them all.” Use these tools to inform judgement, not replace it.

📦 1. Improve Data Quality and Representation

• Use datasets that accurately reflect the diversity of the population served.
• Identify and fill gaps in representation (e.g., age, race, gender, geography).
• Audit for:
  • Class imbalance
  • Skewed labels influenced by social or organizational bias
  • Historical drift in data sources

🛠️ Tactics: - Data augmentation or synthetic data to balance under-represented groups - Example: If only 10% of job applicants are women, over-sample to train a less biased hiring model

🧠 2. Fairness by Design in Modeling

• Incorporate fairness objectives and constraints at training time (e.g., demographic parity, equalized odds)
• Apply mitigation techniques across stages:
  • Pre-processing: Debias word embeddings; suppress proxies for sensitive attributes
  • In-processing: Use fairness-aware loss functions (e.g., adversarial de-biasing, regularization)
  • Post-processing: Adjust prediction thresholds across groups to reduce disparate impact

💡 Note: These techniques involve trade-offs — such as accuracy vs. fairness — which should be documented transparently.

🏛️ 3. Organizational Practices

• Form diverse, cross-functional teams and seek diverse perspectives
• Include affected stakeholders in design reviews (“Nothing about us without us”)
• Establish Ethics review boards or run bias bounty programs

🔍 4. Ongoing Auditing and Monitoring

• Use fairness dashboards and slice analysis tools to monitor deployed models
• Regularly re-audit as user populations or data distributions shift
• Incorporate feedback loops (e.g., from complaints or override events)

📋 5. Transparency and Accountability

• Publish model cards and data datasheets
• Assign model “owners” or responsible parties for long-term oversight

✅ Bias mitigation isn’t a one-off fix — it’s a continuous governance practice.

• ✅ Bias in AI is real and measurable: AI systems can inherit and amplify social, historical, and institutional biases, especially when trained on skewed data or optimized without fairness constraints.
• ✅ Sources of bias are multi-layered: Bias can arise from data, algorithms, design assumptions, or deployment context — and must be addressed holistically.
• ✅ Fairness ≠ one-size-fits-all: There are multiple mathematical definitions of fairness (e.g., demographic parity vs. equalized odds), and they can conflict. Choosing the right one depends on values, context, and trade-offs.
• ✅ Case studies show real-world harm : From wrongful arrests to biased hiring and healthcare disparities, the consequences of unmitigated bias can be legal, ethical, and reputational.
• ✅ Mitigation requires both tech and process: Tools like AIF360, Fairlearn, and What-If Tool are helpful, but human judgment, diverse teams, transparent documentation, and governance are essential.

🎯 Your role as a leader: Drive fairness by asking better questions, allocating resources, and building a culture where ethical AI isn’t optional — it’s the default.