The 8-Minute Binary Search and the Dragon‑Slaying Blade

Ten years ago, I wrote a chat backup tool that stored everything in plain text. One JSON object per line, one message per line. It was simple, fast, and had no dependencies. The kind of design that makes engineers nod in approval. No database. No schema. Just append a line to a file and walk away. I was proud of how little it did.

One evening after work, a colleague leaned over the desk and said, “Help me find my chat with her from March 2, 2017.”

The file was four gigabytes. I opened it, scrolled somewhere into the middle, and read the first timestamp I landed on. November 2019. I jumped forward two hundred lines. Still 2019. Jumped backward roughly half the remaining distance. March 2016. I kept narrowing the window like that, manually, reading one date at a time. It took seven guesses and eight minutes to locate a single day of conversation. Eight minutes to answer one question.

That night I built an index. Nothing fancy. A separate file that mapped each calendar date to a byte offset in the main chat log. The lookup shrank from eight minutes to two seconds. Problem solved. I shipped it, and for a while, it held.

Then came the deletion request. A user had typed the word “hotel” in several chats and now wanted every trace gone. Each deletion shifted all the text after it by some number of bytes. The offsets in my index file were now wrong. Not slightly wrong. Systematically, progressively wrong. Every removed line made every later entry point to the wrong place. I tried to be clever: recompute only the affected ranges, apply the delta to everything downstream. But the index was sequential. After three hundred deletions, seventeen thousand lines needed recalculation. The time cost ballooned back to where it started.

That was the moment I moved the data into SQLite.

Date queries now ran through a B‑Tree. Deleting a single message touched only one page. The rest of the index stayed intact. The difference was not theoretical. A query that had taken minutes on a freshly rebuilt index now completed in milliseconds, no matter how many holes the deletions left. Tagging arrived as a feature request. Encrypting messages per user. Starring individual chats. The product manager dropped each of these on my desk with the same casual tone, and SQLite absorbed them. ALTER TABLE, a new index. I never touched the storage layer. It just worked.

The technical design review was a different kind of test. I walked through the migration plan, the schema, the query patterns. The architecture lead listened, said nothing, and left a single comment in the document. Eight characters: “A dragon‑slaying blade — beware the blood of misuse.”

He had a story. A previous team had used SQLite to store the toggle states of a settings page. On‑off switches. A flat list of booleans. They justified it with the phrase “for future cross‑table analysis.” The table grew over time until it held thirty‑seven fields. The ORM layer alone reached five thousand lines. One schema migration tied up two engineers for two weeks. The project delivered no cross‑table analysis. It just moved slower forever.

That comment haunted me, but not because it was wrong. It was exactly right. The risk of a tool like SQLite isn’t the B‑Tree overhead or the binary size. It’s the person who sees a relational database and immediately starts modeling toggle states as normalized entities. The same mind that builds indexes for future queries that never arrive. The same impulse that turns a chat log into a platform before a single user asks for it.

Now, whenever someone says “chat logs should just be plain text” or “SQLite is too heavy,” two numbers surface in my head. Eight minutes to find one date in a four‑gigabyte text file. Thirty‑seven fields in a settings table that could have been a flat JSON blob. Both are true. Both are warnings. The lesson is not to avoid powerful tools. The lesson is that the danger lives in the justification, not the tool. A dragon‑slaying blade is the right choice when you are standing in front of a dragon. When you are storing the state of a checkbox, you put down the sword and reach for something lighter.