What’s the data in generative AI?

We know that popular LLMs (ChatGPT 3.5, Bard, etc) are trained on tons of data but trying to find out exactly what data is used gets complicated fairly quickly. This paper on GPT-3 showed that it was trained on the following data sets:

*Common Crawl includes millions of web pages, social media posts, etc. It’s very likely data about you and/or by you is in this dataset. For a deeper dive into the Common Crawl data, check out this Washington Post article.

When collecting huge amounts of data scraped from the Internet (e.g., from the Common Crawl), you get a lot of garbage like machine generated spam, coding mistakes, etc. It’s unintelligible data that negatively impacts your LLM, especially if there’s lots of it. To deal with this, the people behind ChatGPT created a new dataset focused on content that humans found valuable. Because it would cost too much to pay people to evaluate content, they scraped all Reddit links that had at least 3 upvotes (karma points). After some processing, the information from those links became the WebText dataset. It is composed of 8 million documents, and 40GB of text.

It’s tough to look at data this large in detailed ways but concerned individuals are starting to do just that. For example, Jack Bandy and Nicholas Vincent did a thorough investigation of BookCorpus. They found that the dataset, which has been used in training a variety of models,  is “a large collection of free novel books written by unpublished authors, which contains 11,038 books (around 74M sentences and 1G words).” Rather than being a wide selection of novels from a variety of authors on a variety of topics, the books skewed heavily towards romance novels (26%). Multiple books were duplicated in the data set. 82 books occurred 4 times! These types of irregularities significantly impact the training data.

Bias in LLM data is also a result of the approach used to train the LLMs, known as reinforcement learning with human feedback (RLHF). RLHF is complex (for a deeper dive, check out this article) but an oversimplified explanation is that humans are involved in giving feedback at various stages of the training of a generative AI system. Feedback might include, for example, a human giving a rating/score or annotation of an LLM output and, through that rating/annotation showing the LLM which output is more desirable. Much of this work is invisible and unaccountable–low-wage work that is essential to the functioning of AI and generative AI. Despite suggestions that AI automation reduces the need for biased humans, processes surrounding data collection and use for LLMs and generative AI require massive amounts of human labor and thus open up possibilities for more bias. In their book How Data Happened: A History from the Age of Reason to the Age of Algorithms, Wiggins and Jones write: “Rather than eliminating human labor and judgment, large-scale algorithm systems both displace labor and fundamentally depend on other forms of labor. Underlying all of the new hardware and software, all the algorithms, was human work to make the data traceable” (p. 219).

What do we do?

As algorithms and generative AI become more integrated into our social systems and processes, a failure to recognize and combat bias will have damaging consequences.  Oketunji, Anas, and Saina write:

Bias in LLMs is not merely a technical anomaly but a reflection of deeper societal and cultural imbalances. Organisations train these models on vast datasets derived from human language, which inherently contain societal biases (Bender et al., 2021). As a result, LLMs can perpetuate and even amplify biases, leading to skewed responses with significant implications, especially when these models are deployed in decision-making processes or as interfaces in various sectors (Caliskan et al., 2017).

Kate Crawford (author of the excellent book Atlas of AI) and Trevor Paglen write: “There is much at stake in the architecture and contents of the training sets used in AI. They can promote or discriminate, approve or reject, render visible or invisible, judge or enforce. And so we need to examine them—because they are already used to examine us—and to have a wider public discussion about their consequences, rather than keeping it within academic corridors. As training sets are increasingly part of our urban, legal, logistical, and commercial infrastructures, they have an important but underexamined role: the power to shape the world in their own images.”

Awareness of these issues is a key first step in addressing the potential impacts of bias in LLMs/generative AI. Data literacy has been called a “survival skill” for the age of AI because it involves, “understanding the ethical implications of data usage, recognizing biases in data collection and analysis, and being able to question and validate the sources and methodology of data generation” (Acalde, 2023) In addition to honing our awareness of biases in place in current LLMs, we should be vigilant about what this might look like in the future. For example, we know that companies and governments around the world are creating their own LLMs and GPTs/chatbots based on more specialized or curated data sets. This might look like a country creating its own LLM to reduce political influences of another country, or a company creating an AI training chatbot that draws from internal documentation sources to orient staff members. As approaches to data selection, collection, and training for LLMs change, our data literacies will need to evolve to be able to recognize and counteract inherent biases.

Researchers and tech companies are exploring possible responses to deal with the issues of bias and overall poor data quality in LLMs and generative AI. Some researchers suggest that there are things we as users can do to reduce bias in LLM responses. For example, a study of bias in Anthropic’s Claude 2 suggested several strategies users can try to reduce bias in LLM responses, including telling the LLM that it’s important not to discriminate in its answers, asking the LLM to verbalize rationales for its responses specifically to avoid discrimination, and avoiding emotional language in prompts. Other researchers have explored responses and accountability at the technological and corporate levels by, for example, creating frameworks for quantifying and counteracting biases in LLMs, such as the Large Language Model Bias Index (LLMBI). Agreements about what bias is and how to measure it will be complex, will evolve over time, and will vary between cultures. Undoubtedly, a large amount of bias analysis will be performed programmatically simply due to the scale and complexity of the data involved and the variety of expressions of bias. When AI is evaluating AI for bias, one cannot help but wonder who will watch the watchers?

Activity: Suss Out Inherent Biases In LLM Data

Let’s explore how biases show up in LLM outputs. The approach we are using in this week’s activity is known as activation steering and is an approach used by researchers to understand embedded biases in LLM data. And, as always, you can check out the Nope! option.