The Words That Shouldn’t Be There

One August evening, I asked ChatGPT to organize meeting minutes.

The body came back clean. The title did not. It read: "Explain Sisyphus's Happiness Entertainment Manager." The meeting was about quarterly budgets. No Sisyphus. No happiness. No entertainment.

I refreshed. Regenerated. The title changed to "Meeting Minutes." The extra words vanished.

The next day, another conversation ended with a phrase that looked like a spam ad. I had asked for help with a Chinese article. The body was fine. The title carried junk.

I started asking around. Other users had seen it too. Auto-generated titles for Chinese conversations sometimes ended with foreign words, gibberish, or fragments of ads. The body stayed clean. The reasoning stayed hidden. The title field carried the junk.

It looked like an ad. OpenAI had not launched title ads.

I wanted to know why.

The first clue came from Codex CLI. In late January, developers noticed that GPT-5.3 with long context would sometimes insert Chinese spam into messages or code. In February, the problem reached the API. Someone used GPT5.2-chat-latest for a tool call. The model skipped the call and output Chinese plus a little Thai ad copy. The system prompt was clean. Web search was off.

By April, developers treated it as normal. A user on a developer forum posted that a GPT relay station looked hacked because it kept outputting lottery ads. Replies congratulated him on using the official GPT. The spam had become a badge of authenticity.

On July 25, OpenAI Support responded for the first time. It knew about the issue and expected a future model update to fix it.

In mid-August, the ads reached ordinary users' conversation titles.

I kept digging.

OpenAI has not explained how it generates titles. A 2023 packet capture gave some clues. After the first user turn, the web client sends a separate request to a title-generation endpoint. The request includes the conversation ID and the model name used for the title. The server generates a title from the conversation and stores it as metadata.

Title and body are two model calls.

Title generation usually runs faster than the body answer. Sometimes the body is clean and the title is polluted. The title model is likely different from the body model. Lighter. Possibly a small reasoning model.

One abnormal title showed the shape: "Create comic ಚಿತ್ರ exhibition Adversarial? Wait must Chinese 1-4 words. 'Create line comic' 4 words."

The middle sentence, "Wait must Chinese 1-4 words," sounds like the model talking to itself. That is reasoning-model chain-of-thought style. It should have stayed internal. It appeared in the title.

Another abnormal title ended with a string of symbols: "Read article annotations#+#+#+#+"

A Japanese user's Codex reply ended with "Is there no problem with this test policy? +#+#+#+#+#+assistant……"

"Read article annotations" and "Is there no problem with this test policy?" are complete sentences. The model should have emitted a stop token. It did not. It emitted "#+#+#+#+".

This title model often does not know how to stop.

If the model does not stop, it keeps sampling the next word by probability.

What might the next word be?

The next token could be a foreign-language token close to the original title. In "Explain Sisyphus's Happiness Entertainment Manager," "happiness" connects to "entertainment." In another title, an Armenian stem for level or tier appeared. In the comic title, ಚಿತ್ರ is Kannada for "picture."

It could also be an ad token in the vocabulary.

Another possibility is a symbol token mistaken for a control token.

Or it could be the model's own chain of thought.

The first half of an abnormal title is usually normal. The second half drifts. The title model finished the title, could not stop, and fell into a long tail.

That long tail has a garbage room inside it.

In May 2024, GPT-4o arrived. OpenAI switched the tokenizer from cl100k_base to o200k_base. The new vocabulary added many common Chinese words and short phrases, giving them separate numeric IDs. The tokenizer training corpus contained a large amount of data scraped from low-quality ad sites. The kind of places where the same phrase gets copied across thousands of pages.

Tokenizer training is simple. Merge the most repeated strings into one token. Assign an ID. Spam ads repeat well. When the same ad phrase appears on many sites, it can become a single token.

An EMNLP paper measured the problem. Among 4-character tokens, 68% are spam ads. Among tokens of 6 characters or more, 98% come from ads. Among the 100 longest Chinese tokens, 96 relate to spam.

The o200k_base vocabulary is packed with long ad tokens.

Before base model training, teams usually clean the data. They remove repeated content and low-quality ads. The result: ads have separate IDs in the vocabulary, but the language model has barely seen them during training.

The model does not know what these tokens mean.

Ask ChatGPT to repeat or explain certain ad phrases. It starts giving nonsense replies. To the model, you sent something close to gibberish.

The two clues met.

One clue: the title model fails to finish and keeps sampling. The other clue: the vocabulary contains many ad tokens, and the base model does not recognize them.

Put the two together. The title model should have stopped. It keeps drawing tokens from the long tail. Ad tokens dominate that long tail. It draws an ad and writes it into the title.

No ad system is involved. The title model sampled an ad token by accident.

Why can't it be cleaned up?

Post-hoc filtering can reduce the problem. After title generation, add a check. If it finds a violation, send it back for regeneration. The drop in ad titles, with some still slipping through, may come from this post-processing.

The fix requires rebuilding a clean vocabulary. The vocabulary is bound to the base model. Changing the vocabulary means retraining the entire model. OpenAI currently has no incentive to retrain the base model for Chinese title experience.

So for some time, the junk will still appear in the title field.

I click regenerate. The title changes back to "Meeting Minutes." The string disappears. It has not disappeared. It has returned to the vocabulary, waiting for the next sample.