Week 05: They’ll know us better than we know ourselves?

Imagine this—you drive towards a large retail store. As you pull into the parking lot, cameras (like these) first log your license plate, then use AI facial recognition software to identify you. Once you enter the store, your exact location is tracked via Bluetooth beacons. AI tools make connections between data collected from your purchase history, credit card statements, social media posts, browsing history, communications with contacts, calendar items, and emails (and much more!) to “predict” that you are visiting the store to buy a present for a friend whose birthday party you are attending next week. You receive a text message from the store: “Looking for a gift for Thalia?” with a link to a coupon for the cute lamp you searched for online last week. As you walk through the store, your every movement (including your gait!) and facial expression is tracked and logged. Did you giggle at the mug in the Kitchen Items section, or touch the fuzzy blanket in the Home section? The digital price tag modifies the price as you approach.

You might read that last paragraph and think, “GREAT! How helpful!” or you might read it and think “Geez, that sounds like a dystopian nightmare.” Or maybe a little of both. Companies that collect gobs of data on us are hoping that our excitement will outweigh being creeped out by how much data they are collecting on us. That’s what is behind hyper-personalization. We are offered bespoke services and experiences in exchange for our data (and our money, of course). You can see this with AI agents; they will manage your calendar, attend your meetings, order your groceries, and all they need is access to your data.

Hyper-personalization is a marketing tool to help companies sell us more things. Beyond that goal (or maybe as an extension of their profit goals), we see companies looking to use data to predict or identify trends to shape our behavior. For example, companies are increasingly interested in what’s called “emotional analytics”–using AI to analyze your tone of voice during customer service calls, your typing patterns when you send messages to friends, even the micro-expressions on your face when you view content online. This “emotional data” is fed into AI systems that attempt to build detailed profiles of people, predicting not just what they might buy, but how they might feel, react, or behave in various situations.

Consider how streaming services don’t just track what shows you watch, but how you watch them. Do you binge episodes late at night? Rewind certain scenes? Pause when specific characters appear? Netflix knows. YouTube tracks it. This viewing behavior, combined with your social media activity, music preferences, and other digital footprints, helps companies create what they call a “taste profile.” But it’s more than just your entertainment preferences being mapped– it’s an attempt to understand your emotional triggers, attention patterns, and decision-making processes. This might feel innocuous when applied to entertainment choices, but what about when applied to crucial areas of our lives like healthcare and education? Health apps and wearable devices collect vast amounts of biometric data– heart rate, sleep patterns, exercise habits, even gait. This information, combined with browsing history and social media activity, could be used to predict potential health issues before they manifest. While this might sound beneficial, it raises questions about who owns this intimate health data and how it might be used beyond medical purposes. Similarly, educational technology platforms track not just what students learn, but how they learn – their attention spans, problem-solving approaches, and emotional responses. This data is used to create “learning profiles” that could follow students throughout their academic careers and beyond.

None of this may feel new to us. Companies have been collecting and collating data on us to better sell us things and/or to shape our behaviors for years. What’s different with the emergence of generative AI and wider use of predictive AI is that companies can use those tools to more quickly, more automatically, and to more accurately target individuals with persuasive personalization techniques (the authors write that, because of generative AI, “one of the most powerful forms of behavioral influence now has the potential to be implemented at scale”). The convergence of all this data – behavioral, emotional, biological, and cognitive – combined with powerful generative AI tools creates something unprecedented. Where earlier systems might have simply categorized us into demographic groups or basic preference clusters, today’s AI can generate highly nuanced, individualized models of each person – what some researchers call our digital twins. These AI-generated profiles don’t just reflect who we are now; they attempt to model how we might think, feel, and behave in future scenarios. They can then run millions of scenarios to refine personalized behavioral models. With generative AI, companies can automatically create thousands of personalized variations of content, products, or experiences, each precisely tailored to trigger specific emotional responses or behaviors.

Beyond the implications for manipulating our purchasing behaviors, there are potentially significant social and societal implications. Matz et al write, “The ability of LLMs to tailor the language of advertisements, news articles or political speeches, creates a dangerous potential to further enmesh people in their own idiosyncratic worlds that are devoid of a shared reality with distinct others.” This fragmentation of shared reality becomes especially concerning when we consider how AI-driven personalization could shape civic life. For example, we know that algorithmic targeting has been used in prior elections to shape voter behaviors, like in 2016 when Cambridge Analytica used Facebook data (and other data they purchased from data brokers) to create psychological profiles of voters and target them with personalized political ads designed to shape their voting choices. With generative AI and more widespread predictive AI use, the danger of voter manipulation (and polarization) is being turned up to 11. Political campaigns are using AI to generate  personalized messages in real-time, each crafted to exploit individual voters’ specific fears, hopes, and biases. Even more concerning, these systems are being used to create synthetic public conversations – AI-generated social media discussions and news commentary that appear organic but are actually designed to create the illusion of widespread support for particular candidates or positions, further eroding voters’ ability to make informed choices based on authentic public discourse.