Motion capture - digital art in cinema

Andy Serkis doesn't have an Oscar. Although films with his participation have already received dozens of awards, largely thanks to him. The thing is that the work of actors who bring characters to life through motion capture devices is somehow considered second-rate, technical, and not related to real art.

The actors, and Serkis in the first place, do not agree with this. They give all the best, and even more than their colleagues. They play with both face and body, like real actors. Their work is not transmitted directly to the screen, but through computer graphics, but is it so different from masks and makeup?

Gradually, the world of cinema begins to accept their point of view, and the term motion capture is now being replaced by another - performance capture, "capture of acting." Or, if you like, "digital acting".

Who was first?

Mocap in cinema arose as a combination of two seemingly unrelated things. The movements of the drawn characters were copied from real people many decades before the advent of computer graphics.

The forerunner of performance capture was rotoscoping, a technology in which frames with live actors are outlined for a cartoon. One of the first such cartoons was the famous Disney Snow White (1937), and Disney characters were created using rotoscoping, as well as almost all the works of Ralph Bakshi.

And quite independently of the cinema, computer technology developed. The first device that did motion capture in the modern sense was developed in the early 1980s by Canadian professor Tom Calvert from Simon Fraser University. He tied a kind of exoskeleton with built-in rheostats to the hands and feet of the experimental subjects, which fixed the bends of the elbows and knees. Calvert needed this technology to study the human body, treat diseases and help choreographers train dancers - he did not even think about cinema.

Of course, the "armor" greatly hampered the movements and made them not too realistic. The problem was solved with the advent of optical systems that tracked the movements of sensors. One of the first such systems, Op-Eye, used a suit with LEDs and several cameras filming the actor from different angles. The program compared the resulting images, drew conclusions about the position of the body in space, and completed the intermediate frames.

The entertainment industry quickly realized what a useful tool they had at their disposal, but at first motion capture was used mainly for "live" performances at exhibitions and TV shows. For example, Nintendo spent a lot of money on motion capture for Mario - but they did not use it in games, but only in advertising.

In the late 1980s and early 1990s, science and the film industry finally met. The first steps towards rapprochement were timid. In science fiction, it was first used by Paul Verhoeven in the cyberpunk action movie Total Recall. Remember the scene where Quaid tries to go through the metal detector on the subway, and his skeleton with a gun is displayed on the monitor? This was the "first attempt" in science fiction and lasted about ten seconds.

But the main thing is that they paid attention to technology. A couple of years later, in Jurassic Park, animators copied the movements of virtual dinosaurs, first from themselves, and then from dolls that were manually controlled. In 1993, Acclaim modeled the characters of the famous fighting game Mortal Kombat from live athletes. In 1999, Arnold Vosloo played the revived mummy of the priest, Imhotep, in the film The Mummy. And a year later, with the help of this technology, they depicted a crowd of spectators in Ridley Scott's Gladiator - each of the three thousand Romans in the Colosseum repeated the movements taken from the same actors.

In the early 2000s, studios had already begun releasing cartoons entirely based on m/c. The first of them was Sinbad: Beyond the Veil of Mists but this unpretentious movie did not enter mass distribution and remained almost unnoticed by anyone. Therefore, now a good half of the publications are sure that the first was the Japanese sci-fi animation movie Final Fantasy: The Spirits Within.

There is a certain justice in this: although the picture came out a year later, against the background of Sinbad it looked amazing. The graphic of Final Fantasy does not seem obsolete, and the characters look almost alive even a decade later.

Three main invaders

Two years after The Spirits Within, a sensation called The Two Towers broke out in the world of cinema. It was the second part of The Lord of the Rings by Peter Jackson that made people talk about motion capture as a fundamentally new type of acting game. We don't dare to say that Serkis "recorded the movements" or "voiced" Gollum. Andy was Gollum, he played a small bald ex-hobbit in the same way that his colleagues played their characters. And, perhaps, outplayed almost everyone with whom he shared the screen.

Before The Lord of the Rings, he was a relatively obscure stage and serial actor, and when he sent Jackson his resume, he was sure that only voice acting would be required from him.

Gollum for many years became a symbol of computer movie characters as a phenomenon. His "rivals" - the elf Dobby from Harry Potter, Yoda from the prequels to Star Wars were completely eclipsed. Gollum received the only MTV film award in history in the nomination "Best Virtual Character" - and personally came to pick it up, "burst into the air" right in the middle of the actor's acceptance speech. Ahead of Andy were more roles of King Kong in the film of the same name of Jackson and Caesar in the new Planet of the Apes - both in complexity comparable to the role of Gollum, and Gollum, again, in Hobbit.

Along with Peter Jackson, another director who helped develop motion capture was Robert Zemeckis. Creator of Back to the Future and Who Framed Roger Rabbit? founded ImageMovers Digital in the late 1990s and invested $5 million to build a digital technology center at the University of Southern California. He used the developments of this center with might and main in his movies.

Robert Zemeckis invested his savings in computer graphics research

It's hard to take the children's animation The Polar Express as a milestone in the history of cinema but from a technical point of view, it was a breakthrough. Zemeckis gathered dozens of professional actors and "put" their characters on them, while his favorite Tom Hanks played as many as five different roles.

Later, Zemeckis created three more such cartoons, the most successful of which was the mythological fantasy, Beowulf. It would be impossible to distinguish it from live shooting if the director did not purposefully emphasize - with unusual colors, and impossible body parts - that we are facing computer graphics. The actors here controlled mostly models of themselves: it is easy to recognize Anthony Hopkins, Angelina Jolie, John Malkovich. But Beowulf was already more of a stylistic experiment - Zemeckis did not offer anything technically new. The laurels of the revolutionary went to another.

As you know, James Cameron makes a revolution in any field that he manages to look at even for a minute, whether it is underwater filming, 3D cameras, or mocap. Most people remember his Avatar as the reason why we now have to wear 3D glasses in movie theaters. But experts know how many other innovations appeared on the set of this film. After Avatar, a new term has taken root in the industry - performance capture, the capture of an acting game.

The special effects masters at Weta Digital, who worked on Avatar, deliberately separated the digitization of movements and facial expressions. The movements were filmed with a remote camera, almost like on location. Thanks to this, the actors had much more freedom, they could perform tricks, interact with the artificial environment specially built on the site.

The captured movements were immediately passed through the Autodesk Motion Builder program and transferred to the model previously entered into the computer. Cameron became the first filmmaker in history to shoot the actors hung with sensors and immediately see the characters and the virtual background on which they act on his monitor. The image differed from the final film only in low detail - after all, the power of the hardware is not unlimited even in the hands of Cameron.

The facial expressions were filmed with the help of special cameras mounted on the heads of the actors. Not sensors were applied to the faces, but the so-called passive markers - reflective points, the movement of which was recorded by the camera, ignoring everything else.

Of the "big three" directors, Cameron has made the fewest films with motion capture, but it seems that he is the one who can change this technology more than others. It is already known that on the set of Avatar2 Cameron is going to surprise the world with an underwater world - the actors will swim in the pool for the first time, giving the characters realistic movements of swimmers.

Double standards

Breaking into sci-fi, mocap has done wonders. Characters that previously could only be drawn came to life on the screen. Motion capture has come a long way from being an assistive technology that only helped make characters' movements and facial expressions more realistic, to a tool that allows actors to turn into their characters. Andy Serkis suggests treating performance capture as "digital makeup".

Serkis has two Saturn awards and several awards from film critics associations, but they even refused to nominate him for an Oscar and other "big" film awards. At the same time, all three movies of Lord of the Rings and King Kong received Oscars for visual effects, Rise of the Planet of the Apes, and The Hobbit were nominated for the same award. Film academics recognized the skill of programmers who digitized Andy Serkis ... but not the actor himself. The main argument against Serkis was that he did not appear on the screen (although he played the young Sméagol live in The Return of the King).

Conservatives have other arguments as well. They remind us that performance capture allows you to "blind" one character from several actors. For example, in Beowulf, the famous Ray Winston was responsible for the facial expressions and voice of the title character, and the young Alan Ritchson served as the body model. It's hard to tell where one actor ends and another one begins.

But aren't there similar examples in feature films? In almost every action movie, the stars are doubled by stuntmen. There are many cases where the role of one actor is voiced by another. In Star Wars bodybuilder David Prowse portrayed the body of Darth Vader, swordsman Bob Anderson replaced him in fights, and actor James Earl Jones gave the villain a powerful voice. With motion capture, it often turns out just the opposite: one actor is given to play several roles. This allows you to save on fees and not overinflate the staff.

The dismissive attitude towards "digital" actors may also be due to the fact that in the past this work was performed by stuntmen, dancers, and martial artists. Their task was not to play a character, but simply to perform complex movements that the animators could not model themselves. But now is not the 90s. Serkis', Stephen Fry's in Burton's Alice in Wonderland, or Zoe Saldana's and Sam Worthington's contribution is not at all like the work of a model.

Transferring movements is not as easy as it might seem. Often, actors have to portray creatures with completely non-human anatomy through motion capture. In John Carter, mocap was done for four-armed Martians, and in The Hobbit, Benedict Cumberbatch had to get used to the role of the dragon Smaug.

Shortly before the premiere of Rise of the Planet of the Apes, Serkis' partner in the film, James Franco, staged a whole campaign for Andy's nomination for the main film award. According to him, Serkis plays "Che Guevara in the guise of a chimpanzee" and his talent helps to reveal important themes of the film - racism, inequality, and revolution. As we already know, he campaigned to no avail. The Academy of Motion Picture Arts and Sciences did not even agree to introduce a separate category for "digital acting". But Andy has already received a more important recognition - the audience. Only specialists remember those who created and implemented motion capture technology (honor and praise them!), but millions of viewers around the world know who turned it into real digital acting. Gollum paved the way for Smaug, the Na'vi, and the orcs from Warcraft, and showed how to breathe soul into cartoon characters.

So it's not Andy Serkis who doesn't have an Oscar. It's the Oscar that doesn't have Andy Serkis.