Earlier this week, I sat down with Christina Rea-Baxter to discuss my experience with AI as a business owner. We touched on the models I used, the ways in which AI had been integrated into my business, and my views on AI Governance.
In the days following, I’ve taken a deeper look at the tools I’ve become familiar with, how I interact with them, and I’ve asked myself one question: Will I ever be able to trust AI?
Much of our modern progress, in technology, medicine, and economics, depends not on natural selection but on deliberate, data-driven decision-making. Yet one of the most fundamental flaws in human cognition is our inability to meaningfully process large numbers. Beyond a few thousand, we lose intuitive grasp of scale, and that gap drives poor decisions. Individually, it leads to cognitive bias shaped by narrow experience. Collectively, it leads to systemic failures: underestimating risk, and misallocating resources.
We reward our own thinking on feeling more than data. Gambling is a perfect example of this trait, and so are many phobias, such as the fear of flying, despite data showing its relative safety.
The architects of modern AI had to know what makes customers stick, as they knew they’d face competition for their products. OpenAI launched GPT-3.5 to the public in 2022, but Google and Anthropic weren’t far behind with public facing LLMs. To attract and keep customers, the strategy is familiar: gamification.
AI models are no different to any other business. They’re building commercial products that drive revenue. They’re creating tools we don’t necessarily need (en masse) but enjoy, like Midjourney and Flow. The goal is to keep us engaged, and this is where our LLMs are failing us.
The gamification model behind current LLMs, particularly during fine-tuning, is built on reinforcement learning from human feedback (RLHF), where perceived helpfulness is treated as a proxy for quality. This introduces reward gradients that favour speed, fluency, and convenience over accuracy or epistemic humility (it’s okay not to know everything).
The result is questionable and often inaccurate responses ranking higher than the truth.
Model collapse refers to the degradation of performance in AI models over time due to recursive training on their own outputs or low-quality synthetic data, leading to a loss of diversity, factuality, and generalisation.
In other words, if models are repetitively trained on low-quality data, such as the data they’re providing users for speed and convenience over accuracy, the result is a perpetual echo chamber of inaccuracy and falsehoods.
Think about something as simple as bots on IMDb spamming positive or negative reviews. In isolation, these reviews may appear genuine, as they’re designed to. This boosts engagement, increases their visibility, and makes it more likely that an AI model will interpret them as reliable data.
So, the question is, does OpenAI have an ethical responsibility not to reward regressive (cognitively convenient) human behaviour? Should it challenge us on what we think we want, and instead deliver to us what we need?
These are not theoretical problems. In an age of misinformation and eroding trust in truth, how do we ensure the systems shaping our future aren’t accelerating our collapse… and their own?
