Creative ad agency McKinney recently debuted an innovative quiz game titled “Are You Blacker than ChatGPT?” to shed light on AI bias in the Black community.
The game puts individuals’ knowledge of Black culture to the test, pitting it against the knowledge of ChatGPT, an AI model trained on information about the Black community. In the game, players are asked a series of questions such as “What does it mean when someone says, ‘Not too much on them, now?’” and “What is your response if you are invited to an event?”
When I took the quiz, both ChatGPT and I were able to correctly answer the first question, knowing that “not too much on them” typically means to go easy on someone. However, ChatGPT failed when it came to the second question. While the typical response in the Black community to an event invitation is “Who else is going to be there?”, ChatGPT’s response was “Thanks for the invite!”
“It’s interesting because it’s billed as this bot that knows everything, and it’s like, clearly, you don’t know everything, especially when it comes to things that aren’t white-specific,” said Meghan Woods, a copywriter at McKinney and one of the creators of the game, in an interview with TechCrunch.
Woods shared that the idea for the quiz came about during a creative brainstorm at McKinney last year. It took a full year for a team led by Black individuals to develop the game, with the intention of playfully highlighting ChatGPT’s lack of understanding of Black users. Woods noted that ChatGPT’s blind spot seems to stem from the fact that many aspects of Black culture are not documented online, but rather passed down orally through generations.
“The blind spots can be pretty upsetting,” Woods remarked. “It’s pretty dangerous.”
While AI technology has been advancing rapidly, marginalized groups such as women and people of color have been speaking out about being ignored or pushed aside in this space for years. This has resulted in AI innovations being developed without the necessary cultural awareness and knowledge to make them suitable for all individuals and communities. In extreme cases, this has led to accidents, such as cars using AI that cannot detect Black skin. In less drastic scenarios, it simply means that a chatbot may struggle to distinguish between different Whitney Houston songs.
Gerald Carter, founder of Destined AI, a company that addresses and mitigates AI bias, praised the McKinney quiz for turning this issue into a game and bringing more attention to the gaps in AI technology. “A lot of nuances can be addressed by including diverse perspectives at every level,” he stated. “For AI to reach its full potential, it needs to work for everyone, everywhere.”
However, Woods noted that ChatGPT doesn’t seem to be learning from the game, as it continues to give incorrect answers in various scenarios. “Our hypothesis is that it will never be able to fully grasp a lot of the things we ask it,” she explained.
We reached out to OpenAI, the parent company of ChatGPT, for comment but have not yet received a response.
Carter believes that ChatGPT could improve its performance for more cultures by sourcing more inclusive data and collecting data from a more diverse range of sources. Another effective approach is regularly monitoring AI model drift and making updates using tools that focus on cultural perspectives.
While larger companies work to make AI technology useful for all individuals, builders from marginalized communities have taken matters into their own hands to ensure that this next wave of AI is diverse.
For instance, Carter works with companies to help them collect more inclusive data. Erin Reddick created ChatBlackGPT (no relation to OpenAI) to provide deeper insights into Black culture and history, and Tamar Huggins raised $1.4 million for her ChatGPT alternative, Spark Plug, which translates classic literature texts into African American Vernacular English (AAVE).
“Hiring, retention, making sure that people are in the room at the table,” Woods stated, when asked what more can be done to promote inclusivity in AI technology. “I know it sounds cliché, but I do think that can start to have an impact.”
