Hallucinating interns

This tweet was making the rounds the other day:

The user presumably asked ChatGPT to do some kind of analysis that required opening a CSV file. The user asks ChatGPT whether it really did open the file or just made up the numbers. ChatGPT coughs up that it didn’t open the file and made up some numbers.

The interaction makes sense colloquially, and I’m not even saying the user was wrong to ask for clarification in this way, but the question he asked is nonsense and I’m choosing to reflect on it for a second.

The problem isn’t that the LLM hallucinated a response and was caught in the act. The problem is that the LLM is not a person and the user is anthropomorphizing a machine which doesn’t even have the capability to respond to this question.

LLMs have no memory and LLMs cannot learn. They are encoded weights in a complex mathematical matrix which together are chained to produce words which we then confer meaning onto. Treating the LLM as an intern who performs a task and produces a result is fine. Taking the next intuitive step of asking the LLM how it went about producing that result is meaningless.

The LLM cannot remember what it did. The LLM doesn’t have a sense of self. The LLM isn’t admitting to taking a shortcut.

When you’re prompting an LLM the machine receives a transcript of the conversation, to date, along with some other information that ChatGPT or whichever provider has chosen to provide it. Then it starts spitting out words based on the transcript and its preprogrammed matrix.

When you ask your follow up question it doesn’t have any context or otherwise build on what happened before, it just does exactly the same thing over again. It receives a transcript and some context, it pushes that through its matrix over and over again until it’s formulated its response.

You’re not getting the intern to admit it was lazy or lied. Rather you got a whole new intern and shoved a stack of papers into their arms and barked you wanted the truth.

That second intern is lying to you just as much as the first one was.

Now that’s not to say the response the user received in that tweet was incorrect. I’d suspect the model never opened the CSV in the first place and just made up numbers and a response.

I’m not even saying this is an incorrect way to use the tool. I’m sure the model providers are building for how their users use the product rather than the way the product works under the hood. That they’re post-training the models to infer some kind of response to this type of interaction which is meaningful and mostly accurate most for the time. Maybe one day they’re even post-train the models to not hallucinate a response and admit it can’t open a CSV upfront - that would be nice.

But it’s just not how this all works. The tweet went viral because people thought it was funny a model cheered on the user (Good catch!) and then admitted to lying, I guess.

But actually it’s a snapshot of a hallucination stacked on another one. Hallucinations squared.

A funhouse of lies which tells us more about ourselves than AI.

(This post is sort of weak but it’s time to crack my knuckles and get writing again. Looking forward to more.)