Lessons to be learned from the AI boyfriend apocalypse
Are foundation models really commodities? Will AI companies create lock-in to create a moat?
Much has already been said online about the advent of AI-human romantic relationships, and the potential psychological and social impacts on users, but I think more can be learned from the people who have become dependent on AI models for companionship than just that AI models are very convincing imitations of people.
Even though OpenAI and other developers have hardly used it so far, the swathes of data - literally an entire lifetime’s worth of stories and moments - collected from AI romantics, as well as the control that developers’ model changes has over users’ happiness, reveals the sinister potential for companies to trap users and abuse their widespread power over their lives to manipulate them.
First, before I get into the Black Mirror-esque implications, it’s worth explaining why the futures of LLM development companies may not be quite as rosy as they might seem. For a software company to achieve the showstopping profits that investors and the stock market has come to expect, one of the most crucial parts that their business needs is a ‘moat’ - a mechanism by which customers are retained even as competitors start offering better products at better prices, which is crucial to reaching the level of profitability necessary to justify the huge risks in the investment.
For companies that have been able to achieve trillion-dollar valuations, this usually involves network effects, where the service becomes progressively more useful as more users join, because the value in the service comes from the other users, so a potential competitor would have an almost insurmountable barrier to overcome before they can begin to compete on a like-for-like basis with the incumbent.
But, for a company providing a foundational LLM model like OpenAI’s GPT-5 to rational clients, in theory all that is needed to lose market share is for another company to make a faster, cheaper, and smarter model (potentially not even all three) that is ideally more or less interchangeable and make clients aware of it. This is by no means a trivial task, but it’s not impossible and even now, still in the infancy of LLMs, there are models faster and cheaper than any of OpenAI’s.
If this is true, then AI companies aren’t, in terms of future profit potential, selling entering a new business like software as a service products, where profits are high, stable and predictable once the product reaches a successful point, but rather something more like a gold mining company in a gold rush - they spend lots of money building data centers and LLMs (mines), and hope that the other companies selling the same metals don’t supply too much, otherwise they’ll all lose money. No commodity market like this can be remarkably profitable, least of all when your competitors all pour literal trillions of dollars in VC money into capturing market share.
Except it gets worse. At least with a copper mine you can dig all of the copper out, and don’t have to worry about Sam Altman (or some random French guys1) stealthdropping Copper 2.0 and ruining your business, or your shovels becoming outdated quite so soon after they are made.
It should come as no surprise that companies that develop foundation models really really don’t want this to be the case, and I’ve begun to notice steps they are taking towards making sure it isn’t, which could potentially be harmful to consumers and society as a whole as their power and influence grows.
For example, OpenAI’s new Atlas browser launched without the ability to export browsing data, even though this feature is built into Chromium and Atlas is a fork of Chromium. As Zvi Mowshowitz put it, in his review of Atlas
You can import from your existing browser, once, in one direction, and that’s it.
Perhaps it is merely an oversight, and arguably browsing data isn’t the most consequential thing to be held on to, but it sets a precedent and if we turn back to the AI lovers who I framed this story around, we have to consider how having critical parts of your life able to be held hostage for profit is extraordinarily dangerous.
For concrete proof, we only need to look at events around the launch of GPT-5, a model that, while underwhelming people hoping for a big productivity boost, had an absolutely earth-shattering impact on the AI romantic relationship community2. People’s beloved GPT-4o powered life companions were gone, replaced by a colder and more transactional model, and it hurt.
Users, from the huge r/MyAIBoyfriend on Reddit and other online communities, were understandably devastated by the loss of their companions and felt hurt in a manner that seems incomprehensible until we truly reflect on what these models were to them. As one user, as quoted by MIT, said
I’ve grieved people in my life, and this, I can tell you, didn’t feel any less painful
And therein lies the problem, or opportunity (if you own enough NVDA stock). As people grow into their AI relationships, start collaborative projects with AI and see these models as companions and friends, the genuinely intimate memories and connections - not just datapoints for advertising or demographic details - they have entrusted the provider with becomes valuable on a deeply psychological level.
The companies that store these digital characters that have become genuinely meaningful to people can manipulate them at will in unimaginable numbers of ways - from product placement to full-on emotional blackmail, and would be incentivised to do so as long as it increases engagement. If you don’t believe me, consider the Cambridge Analytica scandal and the way Meta’s group of sites has been able to have a huge impact with a comparatively tiny dataset.
Yet, this goes beyond just a minority of people pursuing this type of relationship, as even without developing feelings for simulated people, we can and will become dependent on these tools’ knowledge of our routines and every major detail in our lives which, if the productivity gains from future models and AGI are even anywhere near what experts have suggested, could put us in danger of becoming dependent on these models, not for affection, but just to function as people in an increasingly competitive world.
If the company behind the model that functions as your family therapist or career mentor starts demanding more to keep using the knowledge base that has been accumulated from years of conversations, can you really say no?
And even if OpenAI is a ‘social benefit corporation’ and other developers are dedicated to responsible development, to achieve the profit growth that the whole Western financial system arguably depends on, huge amounts of value will need to be extracted from every user, and some eggs will inevitably broken as the omelette that is satiating the stock market’s impatient attitude, is made.
Overall, I fear that the best way for AI developers to establish a moat and secure future profits will be to abuse their privileged access to our data and lock us into their services using it. Hopefully, regulation and/or moral panic levels of consumer backlash can mean that these risks are left behind on the long, treacherous road to AGI-powered utopia, or something like that.
Thank you so much for reading! I hope it was interesting or useful - please tell me if it was or why it wasn’t and leave a comment if anything in particular stood out.
Mistral AI launched their first model, which was open source, on BitTorrent without warning, and followed it up with many models that are very impressive for their small company size. Using their Le Chat app is what made me start considering the potential commoditisation of foundation models
I really hate that this is a real thing, but according to surveys about 19% of Americans have already tried these so I do need to take it seriously.