The Night a Neural Network Invented a Perfect Triangle

At 3 a.m., the training script was still running. Training error had hit zero. Cross-entropy kept dropping.

In the PCA plot, the last-layer features contracted from a messy cloud into three points. The points were equidistant. The classifier’s weight arrows turned and aligned with them. No one told the network to be symmetric. It arranged itself into an equilateral triangle.

This is Neural Collapse.

Papyan, Han, and Donoho named the phenomenon in 2020. A classification network maps input into a feature vector before the final linear classifier, then computes each class’s logit through a linear transformation. Training error reaching zero only means the correct class logit ranks first for every sample. Cross-entropy still cares how far ahead it is. A correct-class probability of 60% and 99.9% both yield a correct prediction, but the latter has a much smaller loss. After all training samples are classified correctly, optimization can continue. The gradient pushes the logit gap between the correct class and the other classes wider, making the correct-class softmax probability closer to 1. This period is the terminal phase. Neural Collapse usually emerges here.

The first change happens within each class. The mean of the features of class c, μ_c, is the class center. Early in training, samples of the same class scatter across a large region of feature space. In the terminal phase, they move closer to μ_c. The ratio of within-class scatter to between-class scatter approaches 0. Samples of the same class almost land on a single point. The “collapse” in Neural Collapse first refers to this.

Once samples of the same class have gathered, look at the relationships among class centers. There are K classes. Compute the mean of all class centers, then write each class center as a centered vector. When classes are balanced and the feature dimension is sufficient, these centered vectors have similar lengths, and the angle between any two different class centers tends to be the same. After normalization, their pairwise inner product approaches:

−1/K−1

Three classes correspond to the three vertices of an equilateral triangle. Four classes correspond to the four vertices of a regular tetrahedron. With more classes, the same structure appears in a (K-1)-dimensional subspace. This is the simplex equiangular tight frame.

Classification margin gives some intuition. After training error hits zero, cross-entropy keeps trying to enlarge the margin of all classes. The classes receive roughly symmetric optimization pressure, and weight decay controls the scale of parameters. If two class centers are too close, they form a smaller classification margin. Spreading all class centers as evenly as possible gives each class a more balanced margin. The optimization process is complex, but it can leave a highly symmetric solution at the class level.

The weights of the final-layer classifier also change. The direction of the weight w_c for class c gradually aligns with the centered class center. After normalization and ignoring the overall scale, the two sets of vectors have approximately the same geometric structure. This is sometimes called self-duality.

When within-class representations are highly concentrated, class centers are approximately equal in length, and classifier weights align with class centers, the final classification rule becomes intuitive. A linear classifier originally compares logits across classes. At this point, its decision approaches the nearest class center: the class center closest to a sample’s feature is more likely to be the predicted class. Within-class contraction, class centers forming a simplex, classifier weights aligning with class centers, and decisions approaching the nearest class center together constitute the classic Neural Collapse.

Reproduction is easy. Three classes, 160 training samples per class, each sample with an independent index. Each index corresponds to a row in a learnable embedding table, which is the sample’s current three-dimensional feature. Class labels are used separately to compute the classification loss and class centers. The features are fed into a three-class linear softmax classifier. Full-batch cross-entropy, a small L2 regularizer, and continued optimization for 3000 steps after training error reaches zero. Save the initial state, the first zero-error state, and the final state. Use the same two-dimensional PCA projection to plot sample points, class centers, and classifier directions.

At step 11, training error drops to 0. From step 11 to step 3000, cross-entropy continues to fall, NC1 continues to fall, and the RMSE of the center Gram matrix relative to the simplex ETF continues to fall. At the end of training, the pairwise cosines of the three centers are close to -0.5, and the center lengths are almost identical. After removing the common offset of the classifier rows, the directional cosine between the weights and the corresponding centers is close to 1. The nearest-class-center rule and the linear classifier make exactly the same predictions on the training samples. With 7 random seeds, the first zero error appears between steps 10 and 19, and the overall trend remains unchanged.

Change the time scale, and the brain stumbles.

Oxford University began teaching in 1096. The Aztec Empire was founded in 1428. When Oxford started holding classes, the Aztecs would not appear for more than three hundred years. Cleopatra died in 30 BCE. The Great Pyramid was built around 2560 BCE. The Moon landing was in 1969. She was about 2530 years from the Great Pyramid and about 1999 years from the Moon landing. She was closer to the Moon landing.

In 1895 in Ohio, there were only two cars that could run on the road. The two cars collided. The probability is so small it sounds like a joke. The Chinese Museum of Writing issued a bounty: crack one oracle-bone character and win 100,000 RMB. About three thousand oracle-bone characters remain undeciphered. Su Shi wrote “Prelude to Water Melody: When Will the Moon Be Bright?” for his younger brother Su Zhe. A lover? No. A sweetheart? No. His younger brother.

Your body is larger than it feels.

The DNA in one human cell is about 2 meters long when stretched out. The human body has about 30 to 37 trillion cells. The total length is 60 to 74 billion kilometers. Earth to the Sun is about 150 million kilometers, about 300 million kilometers round trip. All the DNA in your body is enough to go back and forth to the Sun 200 to 250 times. A GPS satellite clock runs about 38 microseconds faster per day than a ground clock. Without correcting for relativity, positioning drifts about 10 kilometers per day. Venus rotates once in about 243 Earth days and orbits once in about 225 Earth days. A day on Venus is longer than its year. The Sun rises in the west and sets in the east. Most planets in the solar system are named after Roman gods. Uranus is named after the Greek god Uranus. All the gold ever mined, melted into a pure gold cube, would have a side length of only about 21 to 22 meters.

Geography and nature break expectations.

The Sahara is the largest desert. The Badain Jaran Desert is the highest. Bilutu Peak is 1611.009 meters above sea level. A mountain piled up by sand. Egypt is one of the countries with the highest rice yields per unit area in the world. It is dry and rainless, but its paddy irrigation systems are advanced, sunlight is abundant, and grains love sunlight. Catch one of every animal species on Earth and about 20% of the time you will catch a beetle. Insects make up half of all animal species, and beetles make up 40% of insects. Cats cannot taste sweetness. Their sweet-taste receptor gene is broken. Octopuses have three hearts and blue blood. Their main heart stops when they swim, so they prefer crawling. Geoducks are long-lived; if no one eats them, they can easily live over a hundred years. Kiribati is the only country that spans both the Northern and Southern Hemispheres and is crossed by the International Date Line. Stand in yesterday and today at the same time.

Society, law, and psychology hold their own oddities.

In China, illegal debts are not protected by law. You do not have to repay gambling debts. You can refuse to pay for prostitution. For usury, you do not have to pay the excessive interest, but you do have to repay the principal. The portion of private lending interest exceeding four times the one-year LPR at the time the contract is formed is generally not protected. The full name of a high-speed rail “no-seat ticket” is “no-reserved-seat ticket.” It is essentially a seat ticket. If a seat is free, you can sit in it. When the seat holder arrives, you yield. People who pick lottery numbers randomly bet less; people who pick their own numbers bet more. The odds of winning are no different. Doorway effect: you walk into a room and forget what you came for. Psychologists call it an event-boundary effect. A widely shared online dating ranking, usually quoted without a source, puts men with 1 to 2 ex-girlfriends first, single men second, and men with 5 or more ex-girlfriends third. Treat it as a cultural curiosity, not a rule. The carnation is a symbol of Mother’s Day because Anna Jarvis’s mother loved it most during her lifetime. George I was German. His English was poor; he mainly spoke German and could also speak French. The Sanxingdui large bronze mask is one of the largest known bronze masks in the world. The Liaoning Museum was the first museum established in New China; its predecessor was the Northeast Museum, opened on July 7, 1949.

Corrections.

The claim that Indian banknotes lose a zero as they wear out is a joke. Lake Dongting being the world’s oldest freshwater lake is inaccurate. The modern Lake Dongting basin is young; Lake Baikal is about 25 million years old. The claim that George I could not speak English is an exaggeration; his English was poor. Holders of high-speed rail no-seat tickets can sit in empty seats, but must yield when the seat holder arrives. Illegal debts depend on jurisdiction. For usury, the part above the legal cap is generally not supported, but the principal must be repaid.

Turn off the training script. The equilateral triangle is still on the screen. Three points, equidistant. You go to the kitchen to pour water. When you come back, it is still there.