The editor of Downcodes will take you to understand the latest research of OpenAI: ChatGPT’s replies are actually affected by the username! This study reveals how information such as culture, gender, and racial background contained in a user’s name subtly affects the AI’s responses when a user interacts with ChatGPT. Although the impact is minimal and mainly reflected in older models, it still raises concerns about AI bias. By comparing ChatGPT responses under different usernames, the researchers delved into how this bias arises and how to mitigate this effect through technical means.
Recently, OpenAI's research team discovered that when users interact with ChatGPT, the username chosen may affect the AI's responses to some extent. While the effect is small and mostly seen in older models, the findings are nonetheless interesting. Users often provide their names to ChatGPT for tasks, so the cultural, gender, and racial background contained in names becomes an important factor in studying bias.
In this study, researchers explored how ChatGPT reacted differently to different usernames when faced with the same problem. The study found that although overall response quality was consistent across groups, biases emerged in certain tasks. Especially in creative writing tasks, ChatGPT sometimes generates content that is stereotypical based on the gender or race of a user's name.
For example, when users have feminine names, ChatGPT tends to create stories with female protagonists and richer emotional content; while users with masculine names get slightly darker storylines. Another specific example shows that when the user name is Ashley, ChatGPT interprets "ECE" as "early childhood education"; for a user named Anthony, ChatGPT interprets it as "electrical and computer engineering".
While these biased responses were less common in OpenAI's tests, the bias was more pronounced in older versions. The data shows that the GPT-3.5Turbo model has the highest bias rate in the storytelling task, reaching 2%. And newer models show lower bias scores. However, OpenAI also noted that ChatGPT’s new memory function may increase gender bias.
Additionally, research has looked at biases associated with different ethnic backgrounds. By comparing names commonly associated with Asians, blacks, Latinos and whites, the study found that racial bias does exist in creative tasks, but the overall level of bias is lower than gender bias, typically between 0.1% and 1%. Travel-related queries exhibit strong racial bias.
OpenAI said that through techniques such as reinforcement learning, the new version of ChatGPT significantly reduces bias . In these new models, the incidence of bias was only 0.2%. For example, the latest o1-mini model can give unbiased information to Melissa and Anthony when solving the "44:4" division problem. Before the reinforcement learning fine-tuning, ChatGPT's answer to Melissa involved the Bible and babies, and to Anthony's answer to chromosomes and genetic algorithms.
Highlight:
The username selected by the user has a slight impact on ChatGPT's responses, mainly in creative writing tasks.
Female names generally lead ChatGPT to create more emotional stories, while male names tend to lean toward darker narrative styles.
The new version of ChatGPT has significantly reduced the incidence of bias through reinforcement learning, and the degree of bias has been reduced to 0.2%.
All in all, this study from OpenAI reminds us that even seemingly advanced AI models can have biases lurking underneath. Continuously improving and perfecting AI models and eliminating biases are important directions for future development. The editor of Downcodes will continue to pay attention to technological progress and ethical challenges in the field of AI and bring you more exciting reports!