Return of the King

CGNetworks.com
Computer Graphics and Digital Visual Effects
CGNetworks Proudly Presents
The VFX of Lord of The Rings
Return of the King
Words by Daniel P Wade
Images courtesy of New Line
Productions/ ©2003 New Line
Productions. All Rights Reserved
Taking time out from the
production of extra scenes
destined for the Special
Extended DVD Edition of
Return of the King, Joe Letteri,
Weta Digital’s visual effects
supervisor, takes us through
the final instalment of the Lord
of the Rings trilogy.
JOE LETTERI
Letteri worked at ILM starting
in 1991 on Star Trek 6 and was
responsible for the look development
and lighting design for the dinosaurs
in Jurassic Park. Other productions
he worked on at ILM included
The Flintstones, Casper, Mission
Impossible, Star Wars: A New Hope
Special Edition, Jack Frost, Magnolia
and Daylight.
Letteri joined Weta Digital as visual
effects supervisor at the start of the
preproduction for Lord of the Rings
The Two Towers, continuing with
the third film Return of the King.
“The thing that was most intriguing
for me”, explains Letteri, “was the
idea that Peter Jackson was building
this whole world, Middle Earth,
and Gollum was also looming as a
big challenge for The Two Towers.
Ever since Jurassic Park I’d spent
a lot of time on creature and skin
development and Gollum would take
this to the next level.”
Courtesy New Line Productions
THE
EFFECTS LOAD
Bringing the fantasy world of Lord of
the Rings to the big screen required
a substantial amount of visual
effects shots to recreate a wide
range of locations and creatures.
The first instalment, Fellowship
of the Ring featured around 400
effects shots; while The Two Towers
featured almost double the shots
at 775. Following the trend, Return
of the King featured almost double
the amount of shots as the previous
film at around 1,500. “We completed
more shots in the final film than we
did in the first two films combined”,
admits Letteri. “Half of those shots
were done in the last two months
before the deadline”.
With a similar post-production
timeframe to the previous films,
the completion of Return of the
King was a significant achievement.
“The way we did it was with a lot
of preparation”, explains Letteri.
“We have a terrific pipeline at Weta
Digital geared towards creating
large-scale scenes and our really
talented crew know their roles.
The animation department worked
well with the model department.
With creature modelling, everyone
Pg 1
Courtesy New Line Productions
knew how to set things up so
when animated, everything moves
properly. Muscles do the right thing,
textures work with body surfaces,
and shaders work with lighting,
compositing and photography. The
whole system was also set up to
mimic the real world with realworld measurements for scale,
photographic standards, printing,
compositing and colour. All this gave
us a commonality we could all work
towards.”
TEAMWORK
During the peak period of postproduction, around 400 artists were
working at Weta Digital to realise to
1,500 required shots. “We’re set up
so people can work well together;
we can create teams of any size to
get the job done. Sometimes it’s
better to have a small team focusing
on a big task and sometimes you
need a big team focusing on a small
task just to get it through in the
time that you’ve got. So we have a
really flexible environment that lets
us do that. We had a shot that we
had to finish with Gollum falling into
the lava. At one point we had twelve
people working on that shot alone.”
CGNetworks.com
I
Computer Graphics and Digital Visual Effects
The pace of work wasn’t unexpected
according to Letteri: “In the end we
were finalising about 100 shots per
week. The team knew we were going
to have a lot of work to do. It was
the kind of show that as long as we
were still standing, we were going
to try to get it up on screen. No-one
wanted to hold back at that point.”
BUILDING
ON EXPERIENCE
The Lord of the Rings trilogy offered
Weta Digital the opportunity to set
up systems and infrastructure that
were perfected over years of postproduction. “We worked out a lot of
the technology and techniques on
the Two Towers”, explains Letteri.
“We thought going into Return of
the King that we were going to be
in pretty good shape because of
that. There had been years of preproduction and a lot of the assets
were already built. We discovered
that wasn’t the case. We had to
rebuild most of it because we were
doing things in the third film that
those creatures had never been
designed to do. For example, the fell
beast had to be completely rebuilt
because we were getting so much
closer to them. It was the same with
n the end we were finalising
about 100 shots per week.
The team knew we were going
to have a lot of work to do. It
was the kind of show that as
long as we were still standing,
we were going to try to get it
up on screen. No-one wanted
to hold back at that point.
Courtesy New Line Productions
Courtesy New Line Productions
the Mûmakil. Even though we saw
them in film two, in film three they
had to do so much more. We wound
up rebuilding a lot of the assets. It
helped to have gone through it once
before, although that amount of
work still takes time no matter how
you cut it.”
photography of New Zealand. These
unspoiled vistas that look like you
could be in this bygone era. So we
stuck to that as much as possible.”
REALISM
One of the great achievements of
Peter Jackson’s Middle-Earth is that
it is a world similar to our own. Even
though many of the characters and
environments couldn’t exist in our
world, they are made from realworld elements that provide a level
of realism which doesn’t take the
audience out of the story.
“In order to get Middle Earth to look
as realistic as possible we worked
as much as we could with real world
reference”, explains Letteri. “The
first film was a little more fantasyoriented because you were seeing
these things for the first time. When
we got to the Two Towers, it was
more about living in the world.
It was very down and dirty. Very
realistic. We relied a lot on just plate
photography: fantastic location
Pg 2
DIGITAL
ENVIRONMENTS
Though New Zealand seemed
custom-made for the purpose of
realising Peter Jackson’s MiddleEarth, Pelennor Fields, the site of the
major battle in Return of the King
didn’t exist anywhere so couldn’t
be filmed conventionally. “Pieces of
Pelennor Field existed throughout
New Zealand”, explains Letteri,
“but the whole thing didn’t exist
in one location. We started with a
sketch that Alan Lee produced of
the battlefield and surrounds. Peter
Jackson then did a sketch which
plotted the different armies and how
they would move throughout the
battle.”
With these details, Weta Digital was
able to start building a 3D model of
the terrain to accommodate what the
battle needed. “We went all around
New Zealand and started finding
locations to shoot the individual
pieces, like mountains to surround
the battlefield”, explains Letteri.
CGNetworks.com
Computer Graphics and Digital Visual Effects
W
ith just under 200 shots for the major
battle sequences on Pelennor Fields, the
challenge to maintain consistency in details
such as lighting was considerable.
Courtesy New Line Productions
“Where the plate photography didn’t
fit the requirements, we would
digitally match what we had, then
replace it and extend it. For example
to produce the grassy plains of
Pelennor Field, we shot a grassy
location in Twizel on South Island.
The big grassy field only extended so
far before it hit a road, so we took
that and replicated it in 3D. We got
the point where we could replace the
grass and do everything as digital
grass.”
how to light the shots and to make
it work. So you could look at the
live action, figure out the placement
on the battlefield and start figuring
out how to light the scene to make
everything work to drop in together.”
VIRTUAL
CINEMATOGRAPHY
Though real-world elements were
used in Return of the King, many
of those elements were enhanced
or extended digitally to create the
larger digital environments. “We
created 360 degree skies which we
The ultimate goal according to
could use for the battle of Pelennor
Letteri was to have the ability to
Fields”, explains Letteri. “Even
move the camera anywhere in
though we augmented them digitally
Pelennor Fields and be able to
to include the looming cloud of
shoot the scene with the complete
environment surrounding the action: Mordor, there’s still real elements
“If we needed to do a helicopter shot throughout, but it’s all digitally
reassembled.”
or get in close, we could just move
the camera around and not worry
Not surprisingly, there were entire
about it.”
shots like the charging Mûmakil
where there were no real-world
With just under 200 shots for the
elements photographed for it;
major battle sequences on Pelennor
everything was constructed
Fields, the challenge to maintain
digitally. “Though the shot where
consistency in details such as
the two Mûmakil collide into each
lighting was considerable. “Once
other initially included live action
we established what our lighting
elements”, explains Letteri, “in a last
design was going to be for the
minute redesign based on something
various stages of the battle”,
that Peter wanted, it was decided
explains Letteri, “everyone knew
that the fastest way to do that was
E X P O S E 1 On Sale Now
www.ballisticpublishing.com
Pg 3
Courtesy New Line Productions
to do the whole thing digitally;
even the riders that come into the
foreground on the shot are digital.”
The city of Minis Tirith which
Sauron’s armies lay siege to
borders on Pelennor Fields and is
seen dramatically as the Nazgul
riding their fell beasts descend
on the city. The Minas Tirith shots
were conceived as storyboards or
via miniature camera moves on
a “bigature” model of the city. A
combination of the two techniques
were used for the sequence where
the nine fell beasts come out of
the Mordor cloud and dive towards
the city. “With storyboards for the
whole shot”, explains Letteri, “the
Miniatures team had to shoot the
dive down towards Minas Tirith. The
cinematography is a blend between
what we needed to do for the reveal
animation of the shot and then going
down to hook up with the miniature
camera to get the actual moves
down on the “bigature”. We then
built the whole environment out
around it so that the camera could
pan through it and see the whole
environment. The characters you can
see in Minas Tirith during the dive
were all Massive-generated.”
T
CGNetworks.com
he tendency
of new camera
systems to smooth
out camera moves
with their gyroscopic
stabilisers meant that
in some cases, camera
movements had to
be put back into the
action.
Computer Graphics and Digital Visual Effects
Courtesy New Line Productions
LIKE
BEING THERE
The tendency of new camera
systems to smooth out camera
moves with their gyroscopic
stabilisers meant that in some cases,
camera movements had to be put
back into the action.
“The camera movements that were
added back into any of the shots or
added digitally were used for specific
effects”, explains Letteri. “The idea
wasn’t to unsteady people, it was to
provide two kinds of realism. When
you’re in the middle of a battle, you
want to feel like your in there with
the riders and the best way to do
that is to make you feel like you’re
on a horse as well, so you want
to give the camera a little bit of
that movement like you’re moving
forward with it. The other kind of
movement we used was camera
shake for the Mûmakil just to get a
sense of their weight using camera
shake for each of their stomps.
Whenever you see the Mûmakil
contacting the ground, there’d be
a camera shake and the closer to
the camera, the bigger the shake.
One thing we also did was for the
footsteps that were off-screen, we’d
also do the camera shakes so that
CGNetworks
Enriching Computer Graphic Artists
Courtesy New Line Productions
the sound guys would have a cue for
the footsteps coming into screen and
going off again.”
ORC
ARMIES
Like Helm’s Deep in The Two Towers,
Return of the King features a major
set-piece battle in which huge
armies face off against each other
on Pelennor Fields. Massive, the
tool that Weta Digital developed to
choreograph thousands of digital
actors in battle reprises its role in
Return of the King on a truly grand
scale. “We started off with the orc
army which Peter wanted to be
about 200,000 strong”, explains
Letteri, “which was about ten times
more than the orc army at Helm’s
Deep”.
Courtesy New Line Productions
walking, attacking, fighting styles
and essentially their own culture.
So we had to develop that for the
orc army and then come up with
a way to make 200,000 of them.
We implemented a level of detail
scheme in order to do that.”
To dress out the army, other
creatures were also required: “We
knew we’d want to put trolls in
there, but rather than reuse the old
trolls, we redesigned them so we
came up with new trolls for Return of
the King. We also brought back the
Warg riders from The Two Towers
who acted as commanders that you
could see on the battlefield in some
of the wide shots. The siege towers
and the catapults had been designed
as both miniatures and live action
According to Letteri, Jackson decided versions which had to be matched,
so we had digital models of all those
for Return of the King that he
that were dressed in throughout. We
wanted the orcs to be stronger and
more regimented, as opposed to the tried to give Pelennor Fields the real
rag-tag orcs of The Two Towers. “We look of a battlefield with fires and
had a redesign of the orc characters smoke as if there were campfires
and forges; anything that a large
with new costumes, new styles of
army would have on the move just
movement and fighting. This was
important for the Massive characters to give it a sense that they’re really
that we created because each of the trampling the earth and beating
everything down.”
characters has their unique way of
timely news,
hard-hitting interviews, the
best high-end content on
Pg 4
CGNetworks.com
CGNetworks.com
Computer Graphics and Digital Visual Effects
D
igital double horses and
riders needed to stand
up right next to the live action
riders in detail and movement.
“Those were done at the hero
level”.
ROHAN RIDERS
The Rohan Riders led by King
Theoden make a stirring entry into
the battle of Pelennor Fields as they
sweep up against the orc army.
Though digital horses and riders
were used in the Helm’s Deep battle
of The Two Towers, their numbers
were significantly increased in
Return of the King to 6,000 and
they appear closer to camera. “We
had to crank everything up to a
higher level with more details in
the horses, especially their manes
and their tales”, explains Letteri.
“The riders also needed a lot more
detail in their clothing and helmets.
We had to write all new shaders to
accommodate getting closer to them
and also getting them to match the
live action and the digital double
horses and riders.”
The charge of the Rohan riders
required many levels of detail
depending on their proximity to the
camera. Digital double horses and
riders needed to stand up right next
to the live action riders in detail and
movement. “Those were done at the
hero level”, explains Letteri, “with
new shaders for the horse’s skin and
different pieces that the riders were
wearing. It took a long time to be
able get those riders to sit side by
side with the live action riders. On
the far end with the wider shots we
had the Massive horses and riders
that also had to hold up. In most of
the shots you saw a mix of all three.
There would be live action horses
and riders both for foreground and
mid-ground and sometimes even
backgrounds. It was the same
with digital doubles. We had digital
doubles that would be up close,
then we’d have Massive riders in
the mid-ground and background.
“E X P O S E
We got to the point where we could
mix freely between them. We could
put in whatever kind of character we
needed depending on what the shot
called for.”
To make the decision on where to
use each type of element, Weta
Digital pre-visualized the scenes:
“Based on what we had in the previz cut”, explains Letteri, “we’d say
‘Ok we’ll shoot this part live action
and then we’ll dress in the rest of
it with Massive or digital doubles’.
Or if it was going to be too hard
to get live action, we’d do it with a
digital double. We had the freedom
to do whatever it needed because
we knew we could put our digital
doubles close to camera and they
would hold up. It gave us a lot of
flexibility to design the shots around
what the action needed. We didn’t
need to worry about what it was
going to look like.”
is absolutely beautiful! It’s a work of art.”
Paul Lypaczewski, General Manager, Discreet. Executive Vice-President, Autodesk
On Sale Now at www.ballisticpublishing.com
MÛMAKIL
The Mûmakil (huge elephant-like
creatures) reappear in Return of
the King as ominous regiment that
sweeps through the battlefield.
Though they appeared in The
Two Towers, the Mûmakil had to
be redesigned for the third film:
“Even though they were the same
shape and size as in film two,
they needed to work a lot closer
to camera, because we’d get right
on top of them with the Legolas
scene”, explains Letteri. “We did a
complete redesign of the skin, the
skeleton, and the muscles which
were completely rebuilt to handle
the big actions that they were doing.
We also added the dressings like the
face paint and the weapons attached
to their tusks. The big towers on
top were dynamically linked to
the Mûmakil so that the Mûmakil
animation would drive the towers
dynamically. Once we solved the
CGNetworks.com
Computer Graphics and Digital Visual Effects
Courtesy New Line Productions
dynamics, we would put the Harad in
the towers (usually between 30-40
in any basket). Those were typically
done with Massive, although for
close-up shots, we would put in
some keyframed animation if we
needed to see them do something
specific. We had the drivers up on
top of the Mûmakil head which were
sometimes shot as blue screen
elements for close-ups otherwise
they were done as Massive or
keyframe animation.”
According to Letteri, something that
occurred in pre-visualization that
wasn’t storyboarded was the scene
where Legolas single-handedly takes
down the Mûmakil. “As we were
pre-vizing Pelennor Fields, we had
this large-scale battle, but Peter
started looking for things for each
of the heroes to do to be part of the
battle. When Aragorn arrives at the
battle with Legolas and Gimli, you
don’t want the army of the dead
just sweeping through; these guys
are our heroes, so they still needed
to be involved in the battle. Legolas
was the natural choice for doing
something this dramatic because
as we saw in the last film he did an
amazing leap onto the horse, so this
was an extension of that idea. When
he’s confronted with the Mûmakil,
he leaps up onto the Mûmakil and
devises a strategy for taking it
down.”
ARMY
OF THE DEAD
Aragorn, heir to the throne of
Gondor, and his companions venture
along the Paths of the Dead to call
upon a ghost army to fulfil a broken
oath to his ancestors. “The army of
the dead were really based around
their king”, explains Letteri. “What
we needed to do first was figure
how to make the king look most
dramatic. What we settled on was a
multi-layered skull technique. The
king was shot live action and there
were two digital skulls inside of his
head. The first stage was a rotted
flesh and the second stage was
the solid skull. What we wanted to
do was create an animated reveal
technique that we could use to
work with the king’s expressions
emotionally. When he was at his
most menacing, threatening or
angry, you would see it burn through
to the skeleton. Otherwise, the
emotions that would flicker across
his face would go from seeing him,
to seeing the decrepit skin, and at
the most intense, flashing down into
the skull. That was really what we
needed to work on first because that
was the most dramatic performance.
We then designed the effect of
the army as a larger scale version
of that. By the time we’re on the
battlefield at Pelennor Field we have
this invincible army and already
know what they are because we’d
seen them in the drama shots. It
wouldn’t have worked to go back the
other way.”
E X P O S E 1 On Sale Now
www.ballisticpublishing.com
Courtesy New Line Productions
When Aragorn, Legolas, Gimli and
the army of the dead arrive on the
battlefield, the army sweeps over
the orcs destroying them as they go.
“The wide shots were done with a
combination of Massive and particle
system techniques on top to give the
sense of this army churning through
the orcs and leaving destruction
in their wakes”, explains Letteri.
“Peter’s idea was that once they
started fighting, they’d be driven by
bloodlust. So as they come off the
ship and start to attack, they crawl
over each other to get at the orcs
and this sets up a rolling wave
motion where they’re charging
ahead flinging themselves in the
air and attacking as they come
down. As soon as they take out an
orc, they’re replaced by the ghosts
behind them, so the army becomes
relentless which makes them
unstoppable as they get to Minas
Tirith and sweep up into the city.”
As the army of the dead sweep over
the battlefield, they also sweep over
the giant Mûmakil. “The Mûmakil
deaths were keyframed so they were
pantomimed first and the Massivegenerated army of the dead were
choreographed to that action”,
explains Letteri. “The drama was
in the Mûmakil’s death, and not as
much as the army sweeping over it,
so the Mûmakil’s death action was
first and the choreography of the
attack was built around that.”
E
CGNetworks.com
arly on in the Two Towers,
we were still learning how
to deal with a creature like
that. The whole idea of using
sub-surface scattering for skin
turned out to be really tricky to
do with lighting and getting the
proper realism. By the end of
The Two Towers we felt like we
understood the technique.
Computer Graphics and Digital Visual Effects
Courtesy New Line Productions
Courtesy New Line Productions
Courtesy New Line Productions
GOLLUM
Building on the groundbreaking work
in The Two Towers of an entirely
believable digital actor, Gollum
becomes an even more dimensional
character in Return of the King.
“Gollum was an evolutionary step
from what we did on the Two
Towers”, believes Letteri. “At the end
of Two Towers, I was pretty happy
with what we had achieved. Early
on in the Two Towers, we were still
learning how to deal with a creature
like that. The whole idea of using
sub-surface scattering for skin
turned out to be really tricky to do
with lighting and getting the proper
realism. By the end of The Two
Towers we felt like we understood
the technique.”
“For the third film, we put a little
CGTalk.com
Digital Visual Effects Proffesionals
more detail into the pores and
wrinkles of Gollum’s face”, adds
Letteri. “We also worked out a new
system for animating those wrinkles.
We really pushed that in Return of
the King because we knew we had a
lot more close-ups. It was a case of
building on what we had. We didn’t
really have to replace or change too
much and it was great to have two
films to work with Gollum because
we could do so much more over the
course of two films.”
Gollum’s appearance changes
throughout Return of the King: “We
plotted a deliberate course where
we would start off Gollum looking
pretty much like he did at the end of
The Two Towers. As they got farther
into Mordor, he would become more
dirty, decrepit and worn out. He was
largest and,
most active
professional forum
Pg 7
Courtesy New Line Productions
starting to shed a lot of the niceties
that he had and was no longer
pretending to be their friend, openly
trying to manipulate them. In some
ways the masks were dropping off
his face and everything got a little
harsher. What we did to pull that
off was to create several stages of
new makeup of dirt and scratches
and scars so that each stage of the
journey he would look more beatup. As we first see them entering
Mordor, Gollum looks similar to the
way he did in The Two Towers. By
the time they’re climbing the stairs
at Minis Morgul climbing the stairs to
Cirith Ungol, he’s a little more beat
up. Sam and Frodo were the same
way. This was done with the makeup
for them as well, so we used that
as a guide to match what they were
doing every step of the way.
CGTalk.com
dedicated to the
Digital Visual Community
CGNetworks.com
Computer Graphics and Digital Visual Effects
C
hristian Rivers, our art director and pre-viz
director, went out and caught one (Funnelweb Spider) and brought it in. So that was sort
of our model for Shelob.
Courtesy New Line Productions
SHELOB
Peter Jackson envisioned Shelob
in terms of his childhood fear of
funnel-web spiders. “For those
of us that weren’t native to New
Zealand, we didn’t know what he
was talking about”, explains Letteri.
“So we got some pictures from some
books and Christian Rivers, our art
director and pre-viz director, went
out and caught one and brought it
in. So that was sort of our model
for Shelob. There was a design that
was done for Shelob’s head a few
years ago that we built from a scan
of the maquette. But Similar to the
experience we had with Gollum in
Two Towers, we realised as soon
as we put the character into shot,
that it didn’t quite work the way we
thought it would. We went through
several weeks of redesigning
Shelob’s head. Last year with Gollum
we started off with shots of just his
arms and legs until we got his head
working. We did the same thing with
Shelob: ‘Where do we see only the
legs? We can do that first.’
“Gradually as we got the head
working, we could do other shots.
If you look closely, Shelob is not a
spider. Spiders don’t have the kind of
“E X P O S E
mouth we put on her. Spiders don’t
have stingers on their abdomen.
She’s not a spider, but a spiderlike creature. We used all of the
creepiness of a spider and especially
the movements. When we started
animating Shelob we thought she’d
be big and heavy, so you’d have
to use heavy movement to get the
weight, but it didn’t look right. The
animators started playing around
with quick spider-like movements
and it looked great. They got it down
to the point where you could still
maintain the sense of weight that
was a perfect combination. That
quickness made her look deadly and
those were characteristic moves of
spiders, so it looked like Sam was
facing a big hairy spider.”
THE
WETA DIGITAL FUTURE
With the success of realising such an
enormous challenge, Peter Jackson,
Weta Digital and the other Weta
companies have become a major
force in the film industry. With the
final work being completed on the
Special Extended DVD Edition,
reflection on the significance of
the project is natural: “The main
thing we’ve come away with in
is absolutely beautiful! It’s a work of art.”
Paul Lypaczewski, General Manager, Discreet. Executive Vice-President, Autodesk
On Sale Now at www.ballisticpublishing.com
Pg 8
Courtesy New Line Productions
the production of the Lord of the
Rings movies is that anything is
possible”, believes Letteri. “We’ve
gotten pretty fearless, or I should
say Peter has gotten pretty fearless
about throwing things at us and
saying ‘this would be great’. He’d
never really ask us if we could do it.
He’d just say ‘this would be great’,
and we’d figure out a way to do it.
That’s a great way to make a movie;
there’s no holding back any more.”
The next step for Peter Jackson
and Weta Digital is a remake of
the classic King Kong. The Weta
Digital team has shrunk from 400
back to 150 core people working on
the Special Extended DVD Edition.
As production ramps up for King
Kong in 2005, the team is expected
to grow again, with Joe Letteri
continuing as King Kong’s visual
effects supervisor. [CGN]
Words by : Daniel P Wade
All Images courtesy of New Line
Productions and Weta Digital