Cutscenes

Transcription

Cutscenes

Simulation Engines
TDA571|DIT030
Multimedia and scenegraphs
Tommaso Piazza
Monday, November 8, 2010
Administrative stuff
 Student Representatives
 People with orange or red extension
IDC | Interaction Design Collegium
Introduction
 Multimedia is everything about the external appearance of
the game that is not directly related to the real-time
rendering
 More specifically
 Music

MP3 and Ogg audio formats

Contextual event-driven music
 Sound effects


File formats
3D sound

Environmental audio
 Video
 Cutscenes (using video or in-game rendering and scripting)
 Graphical User Interface (GUI) (Covered in another lecture)
Example: Data screens in Doom3
 Doom3 uses a lot of interesting multimedia
 In-game computer screens are interactive and show
animations that help progress the game
 Cutscenes are done in-game with the engine
Fundamentals of digital audio
 Audio is a pressure wave in the air
 Digitally represented by discrete samples
 Each sample represents the amplitude of the
wave at a given time
 The frequency of samples
 Sample rate
 The sample size
 8 or 16 bit mostly
 Number of channels
 1 for mono, 2 for stereo
Sample rates
 Telephone quality
 11kHz, 1 channel
 Radio quality
 22kHz, 1 or 2 channels
 CD quality
 44kHz, 2 channels
 Studio quality
 > 48kHz, 24 bits, up to 24 tracks
 Human ear is sensitive up to 20kHz, human voice
has a spectrum of ~4kHz
 Nyquist-Shannon theorem
Windows WaveOut
 Standard component on all flavors of Windows

Provides basic functionality for playing raw sound data

Easy to use and works on various hardware

Corresponding WaveIn also part of the WaveForm API
 How to use
1. Open the desired sound device with waveOutOpen()
2. Allocate a memory buffer and a WAVEHDR structure
3. Call waveOutPrepareHeader() to prepare for playback
4. Fill in the buffer with sound data
5. Call waveOutWrite() to play it
6. Call waveOutReset() to force the device to relase all buffers
7. Call waveOutUnprepareHeader to release resources
8. Deallocate the buffer memory
9. Calls waveOutClose to release the sound device
 Typically not used in games except for basic fallback

Slow, not optimized for games and restricted to the Windows platform
3D and environmental sound
 Throughout the years sound playback has been improved with
support for
 3D sound (positioning)
 Environmental sound (effects that emulate the size, shape and
properties of the surrounding area)
 Sources
 Physical sound sources. Includes information about 3D position,
velocity and direction
 Listeners
 Global listeners instance. Represents the 3D position and velocity of
the player receiving the sound
 Environment
 Information about the properties of the environment, including its size,
shape and other physical attributes. Often tied to the listener
3D sound – Speaker configuration
 3D sound must take in considerations the
hardware configuration of the listener
 There are many standards setups and formats





Headphones
Stereo systems
Dolby Digital 5.1
DTS
...
 Usually this is abstracted away by the sound
API
Creative EAX
 Creative's API for environmental audio

One of the first with commercial support
 Handles environmental effects

3D is handed off to DirectSound, OpenAL, etc
 3.0 was called EAX Advanced HD, latest version on the X-fi is 5.0 with 128
voices and up to 4 effects on each voice
 Multiple environments

Instead of a single set of environment properties, tied to the listener, the later EAXs support
concurrent environments that affect sound differently
 Morphing

Morphing of environmental properties as the listener moves
 Panning

Advanced control of the movement of sound in a 3D acoustic space
 Reflection

Functionality for simulating reflections and echoes of sound against surfaces in 3D
DirectSound
 The sound component of the DirectX SDK
 Optimized for real-time applications
 Better than WaveOut/In but less hardware support
 DirectSound3D gives support for 3D
positioning and environmental effects
 Eats more CPU but can be hardware accelerated
 Slowly being replaced by Xaudio and XACT
 XP, Vista and the XBOX 360 (part of XNA)
 DirectSound on Vista lacks support for hardware
acceleration
OpenAL
 Designed to be the “OpenGL of audio”
 Provides a standardized API for accessing 3D audio
 Uses extensions for adding future functionality
(same as OpenGL)
 No standardized support for environmental
effects, but available with extensions
OpenAL: Fundamentals
 OpenAL uses a simple model of 3D acoustics
 Three kinds of entities
 Sources
 A source of sound in the world. Linked to one or more buffers
containing sound data. Also contains information about position and
velocity.
 Listener
 An “ear” in the world representing the player, including position and
velocity.
 Buffers
 Memory buffers for the storage of sound date for playback.
Controlled by sources and are never manipulated directly. In many cases
they are transferred to the sound card prior to playback and are thus
immutable.
OpenAL: Buffer creating & loading
 After initializing OpenAL we need to create our
buffers
ALuint buffer;
alGenBuffers(1, &buffer);
 Each buffer is denoted by a handle (integer)
 Fill buffers with data with
alBufferData(buffer, format, data, size, freq);
 OpenAL does not in itself contain loaders
for common file formats
ALUT
 Luckily, OpenAL, like OpenGL, contains a
simple portable API called ALUT with support
for a lot of common tasks
ALenum format;
ALsizei size, freq;
ALboolean loop;
Alvoid *data;
alutLoadWAVFile(“sound.wav”, &format, &data,
&size, &freq, &loop);
OpenAL: Source creation
 To create a source
ALuint source;
alGenSources(1, &source);
 Bind the buffer to a source
alSourcei(source, AL_BUFFER, buffers);
 Set 3D properties
alSourcefv(source, AL_POSITION, sourcePos);
alSourcefv(source, AL_VELOCITY, sourceVel);
alSourcefv(source, AL_DIRECTION,
sourceDir);
OpenAL: Listeners
 Since there can only be one listener, it does
not need to be created.
 Configure its 3D position
alListenerfv(AL_POSITION, listenerPos);
alListenerfv(AL_VELOCITY, listenerVel);
alListenerfv(AL_DIRECTION, listenerDir);
 listenerDir is an array of 6 floats defining both
the “forward” and “up” vectors
OpenAL: Playing sounds
 Playing operations
alSourcePlay(source);
alSourceStop(source);
alSourceRewind(source);
alSourcePause(source);
 To cleanup
alDeleteSources(NUM_SOURCES, source);
alDeleteBuffers(NUM_BUFFERS, buffer);
FMOD
 FMOD is a cross-platform audio library
supporting the same functionality on a wide
range of hardware and platforms
 Includes support for software sound mixing, 3D
audio, music playback, Dolby support, etc
 Free for non-commercial use (no sources)
 $1000-4000 per platform for commercial use
 $100 “shareware/hobbyist” license exists as well
Music
 Sound effects suspend disbelief
 Music helps with immersion
 Extremely important
 Older games used CD audio
 Sometimes even for voiceovers (Loom)
 Gives little opportunity for processing
MP3
 Developed by the Fraunhofer Institute in 1997
 ISO-MPEG Audio Layer 3, MP3
 Non-audible frequency principle
 Sound sources next to other sound sources with higher intensity
are masked. Example: A bird singing next to a running car is hard
to hear and can be eliminated
 Principle of less important signals
 The parts of the sound spectrum less important to fidelity
can be eliminated
 Commonly used in games, but the format is proprietary
and jealously guarded by Fraunhofer, so using it requires
a substantial licensing fee
Ogg Vorbis
 Open source equivalent to MP3
 Freely available
 Arguably better quality sound than
MP3 on equivalent bitrates
 There are free libraries available
for using with Ogg Vorbis
MOD music
 Short for “module”
 Devised for the Amiga computers
 Capable of playing four simultaneous sounds
 Stores both samples and the actual melodies
 Generally smaller than MP3/OGG
 Relatively easy to manipulate
 Change pitch
 Change tempo
 Inject tunes
Event-driven music
 Instead of using a single linear song, the music can be
structured into smaller segments, controlled by events in
the game
 Allows for immersive effects
 One of the first was Michael Z. Land of LucasArts with the
iMUSE system (used in X-Wing)
 Two components are needed
 Control logic

Commands and scripts that control the flow of music, including the transition between
segments
 Segments

Audio segments that can be combined into in-game music
 We want support for both simultaneous and end-to-end
combinations
Event-driven music
Cutscenes
 A common story-telling tool where the story is
progressed by non- or semi-interactive
sequences of events shown to the player
 Good examples: The Wing Commander series
 $12 million budget (WC4), most of it cutscenes
 In the past cutscenes have been pre-rendered
 Nowadays, they are mostly done in the game's
engine in real-time
Cutscenes
Cutscenes
Video
 If they are made from film footage or are pre-rendered animations,
cutscenes are stored in video
 MNG
 Sibling of the PNG-format, but for animations. Good for simple sequences.
 Video for Windows
 Uses AVI-files and can use a multitude of codecs (DivX, Xvid, etc)
 MPEG
 Standard format. MPEG-2 is the DVD-format.
 Quicktime
 Apple's video format. Uses .MOV-files and supports multiple codecs.
 DirectShow (ActiveMovie)
 Microsoft's API for video programming for DirectX
 Smacker and Bink
 Commercial SDK and codec specifically designed for games programming
In-game cutscenes
 Current 3D engines are powerful enough to allow for ingame cutscenes
 Provides for more dynamic control of the contents
 Maintains the immersion of the game
 Some requirements for this
 Powerful scripting engine

Artists must be given tools for scripting entities, cameras, characters, etc
using scripting languages or other tools.
 Facial animation

Cutscenes typically involve dialogue. The engine needs to be capable of
rendering characters in closeup.
 The art of creating real-time rendered films is referred to
as “Machinima”
Facial animation
 Believable facial animation is extremely difficult to achieve
 5 categories according to Watt & Policarpo
 Simple movement of the head
 Simple movement of the eyes
 Complex simultaneous deformation of different parts of the
face to form expressions
 Speech – complex deformation/movement of the lips, jaws
and tongue
 Appropriate gestures particularly of, but not restricted to, arms and
hands
 In practice this is often achieved through a combination of
skeletal animations and morph targets
Tool design
 There are multiple ways to generate in-game cutscene, here
are a few
 Demo editing
 Human “actors” are used to control the characters using regular ingame facilities and the action is recorded. Requires large “cast” and
probably a lot of post-processing.
 In-map scripting
 Designers and artists add the scripted behavior to each map using
the normal level editor. Requires tool support and a strong scripting
language.
 Independent scripting
 In-game scripting tools help the designers to control the cutscene as
if they were directors on a set controlling placement of cameras or
actors.
Uses for in-game cutscenes
 Dialogue
 Allows for natural conversations in the games
 Introduction of plot elements
 Single shots or sequences of shots that introduce new
locations, creatures etc.
 Plot development and mission briefings
 Guide the players towards what they are intended to do in
the next interactive portion of the game (Command &
Conquer Series)
 Introductions and conclusions
 Currently, most games still pre-render this, but it is about
to change
The language of film
 Framing
 The positioning of characters, camera and objects within a shot
 Cinematic shot placement
 Establishing shot

Used to establish the location of a scene
 Wide shot

A shot of all the members of a conversation, to establish their positions
 Long shot

Full body of a character, used to introduce a character
 Medium close-up

Common conversation shot - just above or below the upper chest to above the head
 Close-up

Just above or below the shoulders, used for conversation, adds more weight to the
statements or reactions
The language of film
 Over the shoulder shot

A medium or wider shot that focuses on one character while having part of the other
character in shot
 Two-shot

A closer version of the wide shot that allows some emotion to be shown while
establishing position
 Conversational camerawork
 The line

Between two characters having an active dialogue there lies an invisible line over
which the camera can rarely cross without disorienting the audience and making them
feel uncomfortable
 Talking room

Any action on the part of a character has a certain amount of space on-screen beyond
the point from which that character emerges, to “balance” that action
 Shot sequence
 Non-linear editing of the various shots into a coherent sequence
For more information
 Try out the tools available in Half-life 2
 Check out http://www.machinima.com
Break
http://xkcd.com
Scene graphs
 Scenes could be contained in flat lists, but this
would be inefficient for complex scenes
 Instead we use hierarchical data structures
called scene graphs
 Hierarchical based on spatial locality
 Based on trees
 Polymorphic nodes
 A tree is based on nodes
 One parent
 A number of children
Why scene graphs?
 Locality of reference
 Events of interest to a player tend to occur in the
same spatial region. Scene graphs allow for quick
elimination of peripheral regions.
 Content management
 3D content is typically created and assembled in
pieces by artists and modelers. A hierarchical graph
makes life easier when managing these objects.
 Hierarchical objects
 Many objects are naturally modeled with a hierarchy
(such as humanoid characters)
Why scene graphs
 Persistence
 Hierarchical organization makes saving and
restoring the state of the world simple.
 Heterogeneity
 A polymorphic data structure like a scene graph
promotes heterogeneous node types that allow for
lots of different node types, not just basic geometry
objects.
 While this is also possible in a flat object list, the
spatial nature of the scene graph makes this
approach very natural.
Node representation
 Each node needs the following information
 Transforms
 Linear transformation, usually represented by a matrix
 Bounding volume
 Bounding volume fully containing the geometry of the
node and its children
 Render state
 State flags for the 3D renderer (texture, material, color,
depth tests, etc)
 Animation state
 Time-varying node data
Node type examples











Group
Static 3D geometry
Dynamic 3D geometry
Terrain patch
Skybox/skydome
Spotlight/pointlight
User interface node
Animation node
3D sound source
3D text
...
Node: Transforms
 All nodes should contain their linear
transformation relative to their parent
 Local coordinate system
 Final transformation by concatenating transforms
from root to current node
 How the transformation is contained varies, but
the base representation tends to be 4x4 matrices
 Quaternions
 Separate translation, etc
 May cache the world transformation
Node: Bounding volumes
 In order to exploit spatial locality we need
bounding volumes on all nodes
 A bounding volume is a convex 3D shape
containing all the geometry of the node and
its children
 Can be used for
 Collision detection
 View frustum culling
 Bounding volumes may need to be
recomputed
Node: Render state
 Maintaining render states in a hierarchical
fashion has many benefits
 Lights which only affect certain regions, etc.
 When performing render traversal of a scene
graph, the renderer collects the render state
through simple “push”/”pop” operations
 Can minimize the amount of costly render state
changes
Animation state
 Animation state is node data that changes as a
function of time
 Often with interpolations between various linear
transformations
 Can be accomplished with controllers
 A lot can be animated
 Scale, position, rotation, texture, fog, etc
 Hierarchical animation structures can be useful
 Synchronize animations on characters, etc
Classes of Nodes
 Three classes of nodes in a scene graph
 Grouping nodes
 Internal nodes in the tree with any number of children.
Spatially and semantically collect groups of nodes
 Leaf nodes
 The business end of the scene graph. Usually 3D
primitives and meshes with a visual representation.
 Decorator nodes
 Grouping nodes with only one child, used to invisibly
“decorate” a subtree and are often used for render and
animation state changes.
Grouping nodes
Leaf nodes
Decorator nodes
Traversals
 The purpose of a scene graph is not only to
store contents of a scene
 Update traversal
 Computes world transformation matrices, animation
data, bounding volumes, etc
 Render traversal
 Walks through the scene graph and draws 3D
objects if they are visible
 Pick traversal
 Decides which object a user has clicked on
Update traversal
 The scene graph contains a certain amount of volatile
state that changes as the scene changes
 World transformations
 Bounding volumes
 Animation state
 Most changes only affect a node and its children
 Update traversals are typically performed once per
render cycle
 Recursive traversal for most things
 Bounding volumes are computed on the upward return call
Update traversal
# Update the node N given the world matrix M
function update(N : node, M : matrix) : bool
# Calculate the absolute matrix of this node
Matrix W = N.calcWorldMatrix(M)
bool bv_changed = false
# Visit the children
for each child c in N:
# Update this child
bv_changed |= update(c, W)
# Did the bounding volume change?
if (bv_changed):
# Recompute the bounding volumes of this node
bv_changed = N.calcBoundingVolume()
# Return the result
return bv_changed
Render traversals
# Render the node N using the abstract renderer R
procedure render(N : node, R : renderer)
# Set the current transformation matrix into the renderer
R.loadMatrix(N.getWorldMatrix())
# Set renderer state (store old)
R.pushState()
R.setState(N.getRenderState())
# Check the bounding volume of this child
if (!R.isVisible(c)):
continue
# Render this child
render(c, R)
# Restore old render state
R.popState()
Render traversal
 The most basic scene graph traversal
 Draws primitives according to render state
 Can perform view frustum culling on nodes
 Node entirely in view frustum -> Draw graphics for
all children
 Node partially in view frustum -> Recursively test
children
 Node not in frustum -> Do nothing
Pick traversal
 Used when the user clicks on the screen and
we need to deduce which 3D object has been
clicked on
 Uses the current view frustum and builds a ray
starting from the user's viewpoint and passes
through the viewport at the coordinates clicked
upon
 Recursively test on scene graph and sort by
the distance from the viewport
Pick traversal
# Perform 3D picking of the node N using the ray R
procedure pick(N : node, R : ray, L : node_list)
# Adapt ray to current world matrix
R.setTransform(N.getWorldMatrix())
# Intersect ray with this bounding volume
if (!R.intersects(c)):
continue
# Perform picking for this child
pick(c, R, L)
Existing scene graph APIs
 SGI Performer
 “Industrial-strength” scene graph API developed by Silicon
Graphics for OpenGL.
 OpenSG
 Freely available scene graph API with multiprocessing
support. More geared towards 3D visualization than games,
but possible to use with games.
 Java3D
 3D API for the Java platform
 NVSG
 Nvidia's scene graph. Modern and has strong support for
programmable shaders using CgFX.
Scene graphs in Ogre3D
 Ogre3D supports a simple scene graph hierarchy based
entirely on graphical objects
 Read up on
 Ogre::Node
 Ogre::SceneNode
 Ogre::Entity
 All SceneManagers in Ogre3D contain a root SceneNode
to which all visible objects must be attached
 There are plenty of different scene graphs for Ogre3D
 OctTree
 Paging landscape
 ...
Scene graphs in Ogre3D
...
// Create the node and entity
SceneManager mSceneMgr = mRoot->getSceneManager(ST_EXTERIOR_CLOSE);
SceneNode *node = mSceneMgr->getRootSceneNode()->
createChildSceneNode("Node1");
Entity *entity = mSceneMgr->createEntity("Mesh", "test.mesh");
node->attachObject(entity);
// Transform the node
node->translate(Vector3(10, 0, 10));
node->scale(0.5, 1, 2);
node->yaw(Degree(-90));
...

Cutscenes

Transcription

Similar documents

7.9 zettabytes

DCCC`s Lecture Capture Adventure

the complete writer

Internship Presentation

Fall 2013 Activities for Babies to Teens

Issue 3 - June 7, 2008

HERE - First Church Redwood City

Term 1 Week 4 2010 - St Andrews Primary, Marayong

Weekend Breaks and Holidays

Bachelor Master PhD stuDies