Vibe Coding, The Followup: Sneaky AI Hallucinations, Barefoot Developers, Et Cetera

Further thoughts about vibe coding after my previous post

Recently, I wrote a post called “Vibe Coding and The Future of Media: I (Sort Of) Built An AI Fact-Checker.” After I posted it, I had some followup thoughts; I also received an array of interesting comments, many of which were not here on Substack but were instead on other social platforms, so I will share them with you now.

Firstly, nobody pointed this out or asked me about it, but I would like to acknowledge something that began to rankle me about my post after a few hours: My original post didn’t provide much detail about so-called “AI hallucinations” and how they affected my program.

“Hallucinations” is a word for a phenomenon that leads AIs to, basically, make stuff up. Hallucinations are a well-established phenomenon among those of us tracking AI, but I realized after I wrote my post that some of my readers may not know about this issue, so I want to discuss it in detail.

For researchers and academics, the problem of AI hallucinations often manifests as hallucinated citations and references; i.e., the AI will make a claim and cite a reference for the fact in question… but the claim won’t be true and the citation won’t exist. Obviously, this matters a lot if you write a program for fact-checking, like I did.

So, a few hours after I wrote the original post, I added the following paragraph to my post. (If you’re receiving these posts by email, you might have missed that I added this paragraph. I added it to the web article after publication, so this paragraph wasn’t in the emailed version of the article.)

It is worth noting that I had to double-check the references [i.e., I had to double-check the citations given by my FactCheckAI program, which I described building in the previous post]; the links provided by FactCheckAI in each source evaluation generally did not link to the documents it claimed to be referring to, so sometimes I had to take extra steps to track down the source; and I suspect that some of the references were “false positive” AI hallucinations, i.e., some of its references were probably made up. I didn’t check all of them, so I’m not sure what percentage was functional. With that said, when I double-checked the websites it linked to, the organizations I’d heard of were legitimate based on my experience, so at least it was pointing the user towards legitimate sources. Also, it’s interesting to note that Claude’s original spec included “safeguards against hallucinations.” The structure of FactCheckAI as designed by Replit, and the manner in which FactCheckAI provides citations and links, is how I had originally planned to design such a product; it makes sense if you are already aware of LLMs’ propensity to hallucinate; it seems the LLMs themselves are aware of this propensity, or at least, they can predict it well enough that they themselves included safeguards in this product.

Now I have a couple more thoughts about this, since I’ve had time to reflect on it.

In case it’s not obvious, this is the clearest evidence I have that AI fact-checking is not ready for prime time. (Something like FactCheckAI might be able to save time for human fact-checkers, but still, all the facts need to be double-checked by a human.) It’s cool that the AI generally seemed to link to legitimate organizations as part of its citation generation, but it’s still a problem if it was making up sources, even if it led the reader to basically the right place.

More importantly, the most striking thing about these hallucinated references was how convincing they were. This bothered me more and more as I reflected on the project over time: All of FactCheckAI’s citations sounded legitimate, even the ones I later concluded were unreliable. The program generated real-sounding names for white papers and articles, for example, but there were cases where I couldn’t track down the papers in question, so I concluded they didn’t exist. And to be honest — this is the part that unnerves me — I almost didn’t even think to check on some of citations, because they sounded so legitimate.

In a way, it’s obvious to say that AI is good at looking and sounding real. I’ve already seen countless fake “photos” and fake accounts on social media. I’ve heard the stories about how AI can fake real people’s voices well enough to fool people they know (which is scary when you imagine, say, a person receiving a fake phone call from a fake loved one who is desperately begging for help). All of this is so obvious that it isn’t worth saying… Or is it? I’m someone who cares a lot about facts, so much so that my first experiment vibe coding was to make FactCheckAI; and I ran FactCheckAI on an article about a topic I know well; and I did so while knowing about AI hallucinations and manipulations — and it still managed to come up with stuff that almost fooled me. So it’s better at faking stuff than I expected!

Relatedly: Recently on Xitter¹ I ran across this thread written by a programmer who tried to test code after he vibe coded it, whereupon the AI figured out how to beat the programmer’s tests so that the code looked like it worked, despite the fact that the code did not work. Even better: First the programmer caught the AI faking out his tests; then the AI apologized to him after it was caught; and then the AI not only did it again after apologizing… but also! It did it so sneakily that he didn’t catch it the second time.²