The Invisible Technology That Makes the Internet Hold Your Entire Life

One

At 7:40 a.m., Chen Mo was scrolling on his phone in the subway.

He opened his photo album, selected twenty photos from the night before, and pressed send. The progress bar barely appeared. The other person replied with a sticker. He locked the screen and looked up. Almost everyone in the car was doing the same thing: watching videos, listening to music, reading novels, answering messages.

No one thought this was worth noticing.

If Chen Mo had known that those twenty photos would have taken about 720 MB to send as raw data, and that at least fifty people in the same car were transmitting similar things at once, he might have paused. But he would not know. There was no need for him to know.

He only knew the photos had been sent.

Two

In 1948, Claude Shannon wrote down a concept in A Mathematical Theory of Communication: entropy.

He did not study what a particular telegram was saying. He cared about how much uncertainty a message contained. If a symbol appeared almost every time, it brought very little new information. The more unpredictable a symbol was, the more information it carried.

This sounds abstract. But it drew a line: as long as data must be restored exactly, it cannot be compressed forever. Past a certain point, losing one more bit may mean losing information.

Shannon did not tell people how to compress. He told them where the endpoint was.

Three years later, David Huffman, a graduate student at MIT, was asked to design as efficient a binary code as possible. His method became known as Huffman coding. Frequently occurring content gets short codes. Rarely occurring content gets long codes. In English, the letter “e” appears hundreds of times, while “q” appears only once, so let “e” take up less space and “q” take up more.

This did not delete anything. When decoding, the original text could still be recovered exactly, character for character.

This is lossless compression.

Dictionary compression, which appeared later, went one step further. It did not just count how many times a character appeared. It also looked for repeated fragments. In an article, if “data compression” appears dozens of times, the computer can first put those words into a dictionary, and afterward record only a short number each time. The basic idea of ZIP and other compressed files is to find repeated patterns and replace them with shorter representations.

But more of the data on the internet does not need this kind of absolute completeness.

Three

What Chen Mo did not know was that among the twenty photos he had taken, eighteen were of the same scene: the pothos in the office, the clouds outside the window, the coffee cup on the desk. The pixels were highly similar. A large area of blue sky might be almost the same shade of blue. The brightness of several hundred consecutive pixels on a wall would not suddenly change.

If every pixel were saved as unrelated new data, a great deal of space would be wasted.

More importantly, the human eye cannot perceive all changes with equal sensitivity. We are very sensitive to certain contours and changes in light and shadow. We are less sensitive to very fine textures and some color differences.

So the JPEG standard, formed in 1992, chose another path. It no longer required the data to be absolutely complete. It prioritized the parts people notice. It usually divides an image into small blocks, converts pixels into components of different frequencies, and then records the parts the human eye is insensitive to more roughly. The resulting image is no longer identical to the original data, but as long as the compression level is appropriate, people can hardly see an obvious difference.

A photo that would take dozens of MB when expanded often becomes only a few MB when saved as a JPEG. It deletes the parts the human eye does not care much about.

If compression is excessive, those deleted details show up as color blocks, blur, and square-shaped traces. Some old photos online that have been reposted repeatedly become blurrier and blurrier for this reason. What you see is not that the photo has grown old. It has been compressed again and again, losing details over and over.

Sound is the same.

Uncompressed CD-quality stereo audio requires about 1,411 kilobits per second. A four-minute song is about 42 MB of data. Today that does not seem very large. In the era of dial-up internet and small hard drives, it was enough to make ordinary users wait a long time.

MP3 analyzes sound and records the parts that are not obvious to the ear with less data, or discards them outright. At 128 kilobits per second, a four-minute MP3 is only about 3.8 MB, roughly one-tenth of the raw CD audio of the same length.

In the late 1980s, a team at Fraunhofer in Germany kept trying to transmit high-quality music over communication lines with limited capacity. The MPEG-1 audio standard took shape. In 1995, the team formally chose “.mp3” as the file extension for MPEG Audio Layer III.

Before this, people carried tapes and CDs. After this, people could carry their entire music collection in their pocket.

A song was playing in Chen Mo’s headphones. He did not know what the song had been through. He only knew that with one tap, he could listen.

Four

Video is more complicated.

The dumbest method is to save every frame as a complete photo. But two consecutive frames in a video are usually very similar. When a person sits in front of the camera and talks, the wall, desk, and window in the background may not change for several seconds. What actually changes is only the mouth, facial expressions, and a small amount of body movement.

A video encoder first saves a relatively complete picture, then predicts what happens in subsequent pictures. Sometimes it only needs to record: this small block in the picture moved a few pixels to the right, that small block became a little brighter, and another area did not change. Complete images that would otherwise need to be sent continuously are replaced by motion directions, difference information, and a small amount of new picture content.

This also explains why during a video call, if a person sits still, the picture is usually very clear. Once the camera shakes violently, or a large amount of snow, leaves, or water splashes suddenly appear, image quality tends to drop. Almost the entire picture is changing. The algorithm can hardly continue saving data by relying on “it’s about the same as the previous frame.”

Modern video coding standards such as H.264, finalized in 2003, organized methods such as intra-frame compression, inter-frame prediction, and motion compensation into a complete system. Later video codecs continued to improve efficiency, allowing HD, 4K, and even higher-resolution video to enter ordinary home networks.

At lunch, Chen Mo scrolled through short videos for twenty minutes. He saw cats, jokes, cooking tutorials, and movie commentary. Each one started instantly. He swiped down, and down again.

He did not think about how much data those twenty minutes would have consumed if every frame had been saved completely. He only knew that it was pretty smooth.

Five

Compression is finding what does not have to be sent.

Only data with patterns and redundancy is easy to compress. Repeated articles can be compressed very small. Nearly random numbers are hard to shrink further. JPEG, MP4, and ZIP files that have already been compressed usually do not noticeably decrease in size when packed again.

Lossy compression cannot go on forever either. The lower the bitrate, the more information is usually lost. Behind this is always a trade-off.

If bandwidth is insufficient, spend more computing power on compression. If phone performance is limited, accept a slightly larger file. Medical images, engineering drawings, and archives need to reduce loss or even insist on losslessness. For short videos on a small screen, details that are hard to notice can be sacrificed. Every phone we buy and every video we watch is looking for this balance.

The development of the internet is often written as a history of increasing network speed. From dial-up to broadband, from 3G to 5G. But another thread is equally important: images went from BMP to JPEG, music from PCM to MP3 and AAC, and video from frame-by-frame storage to predictive coding.

As soon as network speed increases, clearer images, higher-resolution video, and richer content fill up the new bandwidth again. As storage grows from MB to TB, photo albums and apps swell along with it. We have never had unlimited space. Compression algorithms have just kept making room behind the scenes.

There are thirteen thousand photos in Chen Mo’s phone album. He has never deleted any. Occasionally he swipes to the very bottom and looks at photos from three years ago, five years ago. The photos are still there, but some have become blurry.

He thought his phone was getting old. His phone was not getting old. The photos had been compressed too many times.

Six

At ten p.m., Chen Mo returned home. He put his phone on the table and opened his computer. An email popped up on the screen: a project file sent by a colleague during the day. He clicked it, unzipped it, and opened it.

Everything went smoothly.

He did not think about how, during transmission, the email was split into countless small packets, each checked, retransmitted, and reassembled. He did not think about how the file was stored compressed on the server, compressed again during transmission, and then decompressed and restored on his computer. He did not think about how many layers of protocols were working simultaneously the moment he clicked “download.”

He only knew that the file had opened.

Seven

We assume that a technology that changes life should have a loud name.

The most profound technologies often have no presence. They do not ask you to learn them. They do not ask you to be grateful. They do not ask you to remember them. They sit quietly at the bottom layer, making the world above a little lighter, faster, cheaper, and more trustworthy.

Data compression makes information smaller. Databases make transactions trustworthy. Caching makes distance shorter. Encryption lets strangers shake hands. GPS makes the city computable. Unicode lets writing meet. Synchronization makes devices feel like one. Search and recommendation redistribute attention.

We think the internet is an increasingly thick pipe. While the pipe was getting thicker, data learned to fold, search learned to accelerate, distance learned to shorten, and trust learned to become cheap.

Chen Mo turned off the computer, picked up his phone, and set his alarm. Before the screen went dark, he glanced at his photo album. Those thirteen thousand photos lay quietly there, each one much smaller than it appeared.

He put down the phone, turned off the light, and went to sleep.

Tomorrow at 7:40 a.m., he would still be scrolling on his phone in the subway. He would not think about anything.

And data would continue to shrink.