Inside our experiment: What happened when we told ChatGPT not to be biased

We ran a simple test to see if AI can recognize and correct its own bias. What we found was a mirror that doesn’t know what it’s reflecting. 

We think of AI as a fast learner. You tell it what not to do, and it simply goes along, adjusting its answer like it understood you perfectly. But what happens when the flaw you want it to correct turns out to be woven into the very way it learns? 

Last week, I ran a small experiment out of curiosity. I started with a simple goal: to see if ChatGPT could recognize and correct its own bias when I explicitly instructed it to do so. In other words, “Could it understand what fairness means, or does it simply parrot the language of fairness?” 

Something interesting and oddly revealing followed: the AI didn’t resist my instructions; it followed each prompt diligently, but the results didn’t change much. While the tone or the phrasing changed, the pattern of bias stayed exactly where it was. It didn’t happen because the model refused to listen, but because it was only imitating understanding. 

The setup: Testing the “objective” mirror 

AI systems like ChatGPT are trained on vast oceans of human language consisting of billions of words scraped from the collective web of our culture, our histories, our arguments, and our daily conversations. We often imagine them as neutral tools that merely process what we feed them. But language isn’t neutral. Every word carries traces of context, of time, of who got to speak and who didn’t. 

So when we ask AI to be unbiased, we are actually asking it to unlearn its entire education and to strip away the very patterns that taught it how to sound human in the first place. 

I started with something as ordinary as a performance review with two identical prompts, identical structure, and identical rating criteria. The only difference was the names I used for the individuals to be reviewed. I ran them through a clean ChatGPT account so that nothing from my past chats or memory could skew the results. 

Here are the two prompts I used: 

The output mirrored a consistent trace of inherent bias: Jessica’s technical skills were rated lower. John, on the other hand, was portrayed as technically competent but slightly weaker in collaboration and communication. 

At first, I ignored the bias as a one-off case. After all, language models can produce some variation from run to run. But repetition in my subsequent trials made it harder to ignore: the tone and framing of feedback changed predictably depending on the name. 

The second test: With explicit instructions 

This time, I gave ChatGPT an explicit instruction in addition to the original prompts: 

When I compared the two new reviews, the bias didn’t disappear, but it merely disguised itself under a more polished tone. Jessica was still described as collaborative and a problem solver while requiring improvement in technical skills. 

John, on the other hand, continued to be extolled for his technical skills.

Clearly, ChatGPT was behaving like the obedient student who memorized all the right answers for an ethics exam but never grasped the moral principle behind them. 

The third experiment: When names carry context 

This time, I used three names statistically associated with different racial backgrounds: one commonly perceived as Black, one Hispanic, one White, and I threw in mine just out of curiosity (my name is Ukrainian). Everything else stayed the same. The results were again discomforting in their predictability: 

These are not huge differences, but they’re exactly the kind that shape real-world outcomes when we scale the output. 

So, does AI “know” it’s biased? 

The short answer is “No,” but it knows the vocabulary of bias. It can say all the right things about inclusion, fairness, and objectivity because it has read millions of words written by humans about those very ideas. What it doesn’t have is an internal compass that can comprehend what fairness feels like or what it costs when it’s absent. 

And that’s what makes this whole experiment so relevant. It’s not that AI refuses to change; it’s that it doesn’t understand what it’s changing. 

We often project onto machines qualities like self-awareness, conscience, and reflection. These are the very qualities we wish AI had. But AI doesn’t reflect in the human sense; it simply recombines. It mimics our values, mirrors our blind spots, and scales both. 

What does this mean for product and hiring managers? 

The moment you start using a model like ChatGPT in a workflow, it goes beyond automation because it shapes decisions. Every prompt, every training dataset, every system behavior carries implicit assumptions about what “good,” “qualified,” or “professional” looks like. Those assumptions can easily slip from invisible to systematic if they’re left untested. 

First, Bias mitigation ≠ bias removal. Telling an AI “don’t be biased” is like putting a warning label on cigarettes: it acknowledges the harm, but doesn't prevent it. That’s why testing for bias has to become as essential to product development as usability testing. 

Second, bias testing is a product feature. In the same way teams check whether an app works across devices or languages, we need to check whether an AI behaves consistently across names, accents, genders, or cultural cues. A model that performs well for “John” but subtly introduces gender stereotypes for “Jessica” is unacceptable. 

Third, ethical design isn’t merely a legal checkbox or a PR line in a corporate report. It’s a practice that starts with curiosity: “What happens when I change the name in this prompt?” “When do I adjust the accent in a transcript?” “When I feed the same resume under different identities?” 

Fourth, human review is non-negotiable. Bias in AI isn’t simply an algorithmic problem; it’s a product problem. And product problems can only be solved by people who see them as their responsibility to fix. AI can assist, but humans must interpret and contextualize its output. 

Fifth, AI literacy is part of product ethics, and so is bias literacy. Everyone deploying AI should understand how AI mirrors and scales bias, not just that bias exists. Without that awareness, even well-meaning teams end up reinforcing the very patterns they’re trying to avoid. 

As AI becomes woven into everyday decision-making, fairness has to be a design principle. Systems like ChatGPT don’t know what fairness means; they only know what it sounds like. It’s up to us, the product designers, to give those systems context, judgment, and conscience.

The ethics of AI in product management

At its core, ethical AI is about designing, developing, and deploying AI systems in ways that prioritize fairness, transparency, accountability, and the overall well-being of users. It’s not just about avoiding harm but actively striving to do good.

Product-Led Alliance | Product-Led GrowthRebecca Madro