Visual effects in John Carter

 http://www.fxguide.com/featured/a-world-away-john-carter/

Edgar Rice Burroughs’ early 1900s tales of a fictional Mars (known as Barsoom) have been brought to the screen with modern day visual effects in John Carter. The film, directed by Pixar’s Andrew Stanton, sees the titular Civil War character, played by Taylor Kitsch, transported to the red planet where he meets the nine feet tall Tharks and all manner of civilizations. Visual effects artists from Double Negative, Cinesite, MPC, Nvisible, Legacy Effects and Halon Entertainment all collaborated on the alien world and its inhabitants.

In this roundtable interview we sit down with the film’s overall VFX supe Peter Chiang, and Double Negative’s head of animation Eamonn Butler and animation supervisor Steve Aplin to discuss the Tharks, Thoates, Woola and the white apes. And look out for more coverage of John Carter in an upcoming fxguidetv with Cinesite about their work for the mile-long walking city of Zodanga.

Planning and prep

Original plate. The actor wore a gray suit and a facial capture camera helmet.

Final shot.

fxg: What kind of research did you have to do to fill out the fantastical environments for Mars?

Peter Chiang (VFX supervisor): Well, we spent two-and-a-half years on the picture – we started in May ’09 – and Andrew Stanton and a series of concept artists including Ryan Church and the production designer Nathan Crowley had already been working with Legacy Effects. They did all the ZBrush sculpts on the characters and created the worlds. So when we arrived on the project there was already a great portfolio of artwork. We went to the Viking images of Mars and referenced rocky terrain and it was used as a springboard. In the end, Andrew wanted to use real locations in Utah and to build sets that resembled Mars but didn’t take it into that barren, rocky, geological landscape – he wanted more interesting architecture of past civilizations.

fxg: One of the things you brought to the screen was some very humanistic qualities to the Tharks – how did you plan for them as characters and what proof of concept or testing did you need to do?

Eamonn Butler (head of animation): One of the first things we did was a test back in 2009. It was really just diving in and creating a small scene with a live action actor and a Thark, using our tools as they were and our pipeline as it was. We were trying to find out what the questions were going to be on the shoot, instead of trying to solve it in one go. We learnt that the Tharks were nine feet tall – they’d been reduced from 15 feet in the books – just so we could actually frame them appropriately with a six foot actor. It was still quite high, so if we had to frame things a certain way then we ended up doing things like animating the muscles in the abdomen so when the Tharks talked you would root the voice there. Up until then, we’d always looked at facial animation as just from the top of the forehead to the chin, in terms of putting in controls for the animators. On this film, we decided that because the characters’ necks are so long and they’re such elegant creatures, that our facial anim started at the top of the head and went down to the chest height.

The context for believability on these Tharks was always going to be when you had a real actor standing right next to one. We had to make sure the Tharks didn’t have a made-up methodology or biology for how those muscles would fire and how those faces would work. So we talked to Andrew about maybe re-working some of the muscles on the Thark design so that they made more sense biologically. We re-worked the sub-structure and muscle form so that they made sense to our riggers and subliminally made sense to the audience, too, because we just didn’t want to have everything be fantastical.

                                                                                        

John Carter meets Tars Tarkas for the first time.

Chiang: Character-wise, as well, we had it set out from the beginning from Andrew that these guys were like a tribe of vikings. They like to fight, they’re aggressive. So we did a number of tests before the shoot to explore that avenue so we could get an idea and visual language about how these guys moved. We showed this to the principal actors playing the Tharks – we showed Willem Dafoe, Samantha Morton, Thomas Haden Church and Polly Walker – who brought so much to the characters themselves which we also able to key off.

Butler: Andrew also allowed the animators to use alternate takes. So every take was made available to us. He would give us his selected take of the actor, but then the animators could look through other takes and pick out nice little details or nuances that might make the scene more appealing. The animators really helped to contribute or augment what the actors had done. The actors were on stilts so their physical performance was a little bit limited. They were on stilts because Andrew really wanted the eyelines to be correct in frame. We always shot with our actors at the right head-height whether they were on stilts or using a backpack with a head extension or on a platform. Part of the sell of making the audience believe the Thark was really there was that Taylor Kitsch was always looking in the right place and connecting with that performance.

Shooting and stereo

Original plate.

Final shot.

fxg: John Carter was shot on film – what were the things you were doing on set to survey the light and the scenes?

Chiang: On set we had to really get the methodology to handle the different settings we were in. We always started off with image-based lighting and HDRIs for every scene. When you get it back and start lighting the characters, there’s always additional things you need to do to enhance certain aspects of the scene. There might be particular parts of the frame that Andrew wanted to highlight, so you’d put a soft key on that. We always put additional spotlights and played around with the ambiance. We also tried to use the clean plates as much as possible too for less clean-up. With that there’s no characters occluding the principal characters so we had to put in a lot of shadows and re-lighting.

fxg: What was the lighting and rendering workflow?

Chiang: We used Maya, RenderMan, Houdini and Nuke for comp’ing.

fxg: Although this is a post-conversion film, how did stereo impact on the animation?

Steve Aplin (animation supervisor): Traditionally in a VFX movie where you’re adding in characters in a 2D screening, you can kind of cheat your characters a little bit in space to get a better silhouette and just to read from camera. But we knew going in with the stereo that the CG characters had to be in the exact same space as the on-set performers were. If we didn’t, then the stereo shows it up, so we couldn’t cheat any poses.

Butler: What compounded the problem for us was using anamorphic. We were using the entire plate to compose your characters, which gives you very little room to re-work the plate to add in a digital extension or a Thark. So we had to be very careful when we shot the movie to make sure the DP knew exactly where to look. After the fact, it gave us less wiggle room with augmenting the action or staging the scene. The stereo and anamorphic didn’t make it impossible, but it was like wearing soft, furry handcuffs.

Chiang: When they did decide to do post-dimensionalization on the film, we started to do a lot of lidar on set to help us respect the 3D space and know the volume. Scenes that you normally wouldn’t worry about, you started to think about the left eye a lot more and what needed to be done.

On-set facial capture

On-set plate.

Final shot.

fxg: What was the tech behind the facial motion capture on set?

Aplin: We had a dual camera set-up so that the principal actors would wear a head harness that had two NTSC cameras pointing at their faces to capture the left and right side. We had markers on the actors’ faces, so we could take that stream of video back to Dneg and we had people here tracking it to go onto a digital version of the actor. Then we wrote a transferring tool which could move that over to our Thark face which would give a similar performance to what we had captured on set. From there we were able to animate over the top or just embellish as we needed to. We also found that in some of the close-up work we needed a little bit more fidelity in the face. We wanted to read more of the musculature underneath the skin, so we came up with this idea of trigger shapes, which worked for a number of measurements and compressions from frame to frame. It would fire these pre-made tension shapes of tendons underneath the skin, firing just before a major shape was dialed in. So if you were going to move the corner of your mouth back, you’d suddenly see a flex shape happen underneath the skin. On top of that, we’d run a face skin sim, which would add a little bit of extra jiggle and tie everything together and make it look very solid close up.

fxg: Did you do a separate motion capture shoot for the actors at all, or how did you approach keyframing them?

Butler: The actors were always on set and we’d also shoot a clean pass without them there to give us the freedom later on to add in the Tharks and move them around for composition reasons. But other than facial capture we didn’t capture the actors’ motion on set. We were outdoors for most of the shoot and we didn’t want to get in the way of them performing. Often times, using an optical set-up can be time consuming. We did place a ton of witness cameras on set so we were able to track their position if necessary. But most of the time Andrew wanted to explore the performance the actor gave and augment it slightly or make it more Thark-like.

Watch some behind the scenes footage from on set.

We reached a point where we thought it might be cool for the animators to capture their own performance and feather that in to what the actor did or use it as reference – much the same way that traditional animators use mirrors to look at their faces. So we invested in a Xsens MVN suit and we had a special room at the studio where animators could put the suit on and act stuff out and have it on their desk by the afternoon. Then they could select different takes of their own performance and combine it with what the actors gave us. We ended up using the suit to generate lots of the library animation that we needed to build up a crowd system. We produced over 800 cycles of animation for that crowd library which was more than there were shots in the film that we animated on.

Animators typically don’t like working with motion capture really, but we empowered them to use it for themselves to be able to try out different takes and get out all the bad ideas really quickly and get in the good ideas. And the other thing was we didn’t want to do say mo-cap of the actors on stilts because it looks just like that – actors with leg extensions – and we didn’t want to kill our creative time.

The animation pipeline

Original plate.

Final shot.

fxg: This would have to be one of Dneg’s biggest creature shows – what tools did you use for the animation, crowds and the creature pipeline?

Butler: The proof of concept test we did at the beginning highlighted the need to have a strong rigging pipeline. In the past, rigging has generally been engineering an envelope to move appropriately and then adding fur or cloth to it. In this case, we needed something a lot more sophisticated. So we had LOD (levels of detail) rigs. The lowest level of detail was in our layout phase, the lightest rig. Then the animators had simple rigs eliminating high detail. But they needed higher detail for facial. We created a whole new department called CFX, short for creature effects, and they developed skin sliding, tendons that could be fired automatically or by hand, a muscle-based PSD system to accurately represent what muscles do under the skin. We used a lot of displacement detail with dynamic wrinkles, particularly around the face. We created another department called Shot-sculpt which were used for extreme shots – say when one of the characters is being punished. Those poses in the rig literally broke the rig, but Andrew was very keen to make her look like she was suffering, so we had to push her into a place that was quite extreme. So Shot-sculpt went in and re-worked the rig into something that was incredibly muscular and compensate for the natural limitations you often find in CG rigging. We also developed some crowd tools to allow us to put up to 5,000 Tharks in one scene, say for the ape arena. We used a multi-layered system for that.

Aplin: There was also the whole look development process, right through from the texturing of the skin to the sub-surface scattering. We did a terrific amount of research into relating it back to human movement and skin. There’s a subconscious thing where people perceive it to be more real if they can relate it to humans. Andrew’s idea of having actors on set was a great one because that was really where everything started. I think synergy between a martian and a human was important. For instance, the eyes started off with black scleras and very contrasty irises, but in the end Andrew went for a more white sclera and blue irises. So the function of the eyes, all the meniscus, all the bump and sclera and wet look, the caustics – all needed to develop to resemble a real human eye to give that emotion. It was all about retaining the real-life performance of the actors and translate that.

See some of the animators at work on John Carter.

fxg: The other large component of course was the extra materials on the Tharks’ bodies – leather, fur, stones. How were these done?

Chiang: There were a lot of props – we broke it down into stages. We’d matchmove the first shot, and there’d be a layer of animation to get the performance Andrew really wanted. So we had to construct the pipeline in such a way that sim’ing and all the cloth and creature effects weren’t wasted on shots before they were final’d. So all the caches for the animation were done at an approved stage, and that was brilliant for Andrew because he was used to signing off on shots very early through his animation of feature films. What that meant was we could be very constructive about how and when we would sim cloth and creature effects later. But there was a lot of high-level detail put into the props for both the hero characters and the crowds. In the pipeline we had almost check-boxes which let you add props and turn on and turn off to whatever the scenes demanded.

Aplin: I’d say animation was fortunate in that we didn’t have to get involved in the cloth side of stuff too much. If we got the character into a specific pose that we knew would be problematic for cloth down the pipe, we had limited controls on some of the props hanging off and compose them in a way that would help the sim later on.

Butler: Some of the costumes were actually made for real. In the beginning, Andrew had commissioned full-size macquettes of the hero Tharks. The costume designer actually went and made Tars’ costume with a mantle and so forth, so we had really good reference to start from.

Original plate with buggies.

Final Thoates shot.

fxg: The Thoates which the Tharks use for transport, were filmed with stand-in buggies. Can you talk about how they worked on set?

Butler: Usually animation happens after principal photography – in this case we knew the Thoates had to be animated before principal in order to educate the building of a rig that we could use on set. So we did a whole bunch of walks and trots and runs at different speeds. We extracted the saddle information from that animation and calculated how much rotation and angles and speed. Then we passed that information onto (special effects supervisor) Chris Corbould and (special effects coordinator) Scott Fisher, who went and built four rigs for use on set – only three were on set at one time. They stripped out all of the petrol engines and put in electric engines to be able to record sound. We painted them out and retained just the saddles and added our animation back underneath.

Chiang: What was great about that was it went back to what Andrew wanted on set – to retain the actor’s eyelines and respect the 3D space the creatures would occupy. Having these great big units on set enabled the actors to really feel like they were on a beast and convey that – it made the job a lot easier to frame the shot and get a better composition of what it would look like in the final film. They could go up to certain speeds – we always knew the Thoate run would go up to 40 miles per hour.

Butler: We also did explore potentially using camels on set at one point. Andrew was very excited about it – we even went and motion captured a camel after making a special suit for it. It was covered in tracking markers – he’s quite a star in camel circles now – he looked like a ninja, it was incredible. But he was deemed to be a little bit risky for our actors on set and they can be slightly impractical as well. But we ended up taking a lot of cues from a camel’s motion in the way we animated the Thoates, so it was still really useful.

Man’s best friend on Mars

Woola's proxy on set.

fxg: Woola – the dog-like character – seems to have had a big reaction with the audience. How did you approach him and how did you stay away from it being too cartoony even though it’s a source of humor and has that ultra-speed?

Butler: I think our initial take was actually more cartoony. Here you had a ten-legged creature that can move at the speed of sound. So the animators went naturally to a slightly more caricatured take on it. Andrew kept calling it the Road Runner too. But we had an epiphany during a sequence we called ‘Honeymoon’s over’ which was the last sequence before he returns to Earth. He’s talking to Woola and he takes his medallion and throws it into the ravine. As he’s talking to Woola, our initial takes had him listening and responding like he understood him. Andrew asked us to simplify it, so we took all that out and made it just like your own cat or dog which looks at you. It doesn’t have a clue what you’re saying, it just looks at you with love and affection. When we started holding back on that, and just treating it like a real animal, it started to click and people would read into that performance.

Aplin: We had to play up the vacancy – that was where we got the most laughs, certainly internally. We realized we could make Woola very endearing without doing much by giving him a vacant expression – like a bulldog which we used for reference of his movement. One of the physical challenges with the Road Runner elements, was moving a 400-500 pound creature at such a speed and having it as a believable thing in that world was always going to be difficult. We really had to sell that idea on the back-end when he stopped, so we explored a lot of different ideas for how he could displace his weight when he stops moving so that you actually buy it.

Butler: We explored playing with blur a little bit, so could see the mechanics of his feet, but it just didn’t look like it was in the real plate, so what helped us sell the speed was interaction and dust pick-ups. The big rooster tail of dirt behind him was what showed his strength and speed in the real world.

fxg: For the more intimate scenes with Woola, what kind of stand-in was there on set?

Butler: Initially, Andrew actually made us audition the animators and we sent him video of these. We built a big cardboard box head with handles on the back. So all the animators who wanted to do Woola for the movie videotaped themselves with someone else, say, throwing a stick. We showed them to Andrew and he felt like it was good but he really wanted a stronger performance. Animators are typically kind of shy, so we ended up going with a puppeteer called Todd Jones who had worked at Hensons. He did a phenomenal job. We built for him a giant foam head with some simple puppeteering tools to blink the eyes and move left and right. He would walk through the scene with Taylor Kitsch to help with eyelines and use that as a jumping off point. We also had an animation lead back at Dneg, Craig Bardsley, who was a great compliment to Todd. Craig was very good at naturalistic animation so he kept everything very real for us.

Watch a scene with Woola.

Chiang: I have to say Eamonn and Steve made excellent Woola’s too. When Todd wasn’t available they dressed up in the greenscreen suits.

Aplin: We didn’t have to, we just chose too…

Chiang: So there are some great outtakes. Actually watching the plates is just hilarious seeing them before the animation gets final’d, because you see them just running around with this head. And I believe you took a tumble on one of them Eamonn?

Butler: I did, I took a tumble on my head.

Chiang: Eamonn had to run at 200 miles an hour and obviously couldn’t. It’s all captured on camera, though.

White apes arena fight

Original arena plate.

Final shot.

fxg: How did you plan for the white apes fight and how was it shot?

Chiang: We marked out the volume and there were restrictions in terms of how much greenscreen we could have. We came up with this idea for containers – very large 40 foot containers that you could stack up. It was out in the open in Utah the location. So they were boarded green. Nathan Crowley the production designer provided us with the dungeon entrance up one end and the stand for Tal at the other, so that there was some reference to eyelines. Obviously the ground was the sandy location.

We had a physical prop that Scott Fisher had made as a prosthetic ape head that was on a crane rig driven by an electrical buggy. Then there was an ape kite that acted as a stand-in prop. To build a physical foam white ape is impractical, so we came up with this idea of making kites – very lightweight cut-outs of the characters. So when we took it into post there was some scale reference. Then there were multiple cranes in order to throw Taylor around that Tom Struthers the stunt coordinator had designed. Then there was a lot of digital takeovers, set extensions and CG white apes and crowds of Tharks…

Aplin: Andrew had previsualized that whole sequence extensively with Halon, which nailed down a lot of his story ideas. When you’re previs’ing you can cheat quite a bit and change the size of the apes for better compositing, but of course we couldn’t do that, especially with the stereo. So we did have to re-work a number of the cameras and compositions.

Butler: One of the challenges was the geography of the ape arena which was almost perfectly round. The sequence started out longer and more elaborate – it was one of the first to go into production for us and one of the last to finish – mainly because there was every trick in the book. There were lots of short shots, crowds, interaction, fur – everything.

Chiang: We also came up with a particle sand solver. When you shoot the real sand and you need to dimensionalize it, we really had to get rid of all the sand that Carter was flicking up and replace with CG sand, to create the other eye.

fxg: With the white apes themselves, what was the brief for their design?

Butler: Andrew would say to us they’re the fiercest creatures on Mars. They’re blind, they live in caves. Every time they were on screen, Andrew had this phrase – ‘just push it to 11′ – obviously a Spinal Tap fan. He just wanted to make it clear that these things were all about destruction and he wanted you to be afraid the moment you saw them. So when the second ape is released, they just immediately attack each other. The blind feature helped us elaborate on how they would be able to find their prey by smelling and sniffing and scratching at the ground. It was also a dramatic technique to slow down the action – it gave Carter a chance to figure out how to escape and defeat the creatures.

fxg: What was the fur solution for the white apes?

Butler: Our grooming tools have been developing since 10,000 B.C. and we also had sand falling into it clumping up. And each strand of fur was four feet long so we had to groom them per shot to make it look believable and behave in a dynamic way. It just required a lot of tender love and care to make it work.

Chiang: Because they were jumping around a lot, Andrew wanted to retain the whiteness within the red arena. Every time the apes fell on the ground, he wanted the idea that the sand would get in, but obviously get redder and redder. They would get up and shake off the sand and go white again. So there was a constant re-generation of the texture in order to retain the whiteness of it.

Original plate.

Final shot.

fxg: How did you approach the digi-doubles for Carter in this scene?

Chiang: We’ve increased the photogrammetry process for that, so we do a lot of image-based modeling rather than just cyberscans. We have a lot of photographs shot at different ranges so we can re-light using the existing textures, rather than having to go in and create separate shaders.

Aplin: There were also some key expressions scanned for Taylor Kitsch which we could use for more distant shots. We could transfer the scans to quite simple blend shapes that were broken up a little bit so we could off-set timings and still get a reasonable performance.

fxg: Can you talk about the compositing challenges for the arena?

Chiang: Well it was a combination of plates that made up the composite. There was a location called Rainbow Rocks in Arizona that served as a template for the look of the arena. It’s some natural geometry formed into a horseshoe shape. We flew around that in a helicopter taking photographs for textures which served the basis for the main wall. Nathan Crowley built certain sections, but we did replace most of it rather than blend pieces. So a lot of the time we would take Carter out of the scene and then do a complete re-build of the background, using projections but also cheating the volumetric a little bit so that the Tharks would still read green. The DP Dan Mindel loved to shoot backlit so the ambient became very important on the textures and the way we lit the whole arena. We would shoot one way in the morning and when the sun came around we’d shoot the other way. Then we would need to merge those. It meant that a lot of the live action set became very flat and ambient. When Andrew looked at those shots sometimes he would want more definition and we would go to a complete CG re-build.

An animated director

Director Andrew Stanton and actor Willem Dafoe.

fxg: What kind of sensibilities do you think Andrew Stanton, with his history in animation, brought to the set and to the visual effects process?

Butler: I think there might have been some concern about how Andrew would be on set – you have to make a lot of decisions quickly, it’s very fast-paced and high tension – whereas feature animation can take years and years. But he was really in his element on set and people stopped worrying about it very quickly. He’s also an actor – he took classes as an actor a long time – and he connected with Taylor and everyone else. I think he really impressed a lot of people with how mature he was on set. That really comes from Pixar where they’re very story-driven and budget conscious. They’re one of the most successful studios in the world that just happens to work in animation. I think people think animation is a completely different thing but really Andrew is a filmmaker. We were always in safe hands with him and he could always give an answer or make a creative decision about how something would push the story forward and why it was even in the movie in the first place. And he’d dabbled a little with a live-action DP, Roger Deakins, on Wall-E and he tried to stick to real camera rules. So he already had a language for what he wanted to see.

Aplin: He was also very keen to get one-on-one face time with the animation crew. It was almost like going back to school. He just had an incredible eye and could guide you back on target. He loved to get up in front of us and act out a certain part. His energy is also incredible.

Chiang: I think it was a natural step for him to go into live action – I think he was ready for it. The challenges that a live action feature brought really required the same attitude for animation. He could easily dissect what was going on and methodically work out the best route using the tools we were given. So although we were introducing him to photorealism and getting all the photography right and those technical aspects, he had this ability to take all that on and work it out for himself and really educate us about all the information through all the heads of department. That really made all the working process with him such a joy.