Who`s in the Picture?

Who’s in the Picture?
Tamara L. Berg
Cse595 Words & Pictures
Face Recognition Datasets
PIE database
Yale Face Database
M2VTS
What the world really looks like
(scenes from an airport)
Challenging Even Up Close
Face Scanner Failed -- Wired News 5/16/02
A More Realistic Dataset
Actress Winona Ryder (news) reacts to
remarks by prosecutor Ann Rundle during the
sentencing hearing in her felony shoplifting case
Friday, Dec. 6, 2002 at the Beverly Hills, Calif.,
courthouse. At right is Ryder's attorney Mark
Geragos. Ryder was sentenced to three years of
probation and was ordered to perform 480 hours
of community service. (AP
Photo/Steve Grayson, POOL)
France's Amelie Mauresmo has pulled
out of the Australian Open (news web sites) because of a knee injury. The
French number one, ranked sixth in the
world, has been struggling with a
painful knee cap for two months and
said on December 18, 2002 she would
not be fit in time to resume playing in
January. Mauresmo is shown during the
Kremlin Cup, Oct. 4. (Grigory Dukor/
Reuters)
Dataset (cont)
Doctor Nikola shows a fork that was
removed from an Israeli woman who
swallowed it while trying to catch a bug
that flew in to her mouth, in Poriah Hospital
northern Israel July 10, 2003. Doctors
performed emergency surgery and removed
the fork. (Reuters)
President George W. Bush waves as he leaves
the White House for a day trip to North
Carolina, July 25, 2002. A White House
spokesman said that Bush would be compelled
to veto Senate legislation creating a new
department of homeland security unless
changes are made. (Kevin Lamarque/Reuters)
Dataset (cont.)
Actor Arnold Schwarzenegger
discusses his plans for his campaign with
reporters after he announced that he will
be a candidate for governor of California
in the recall election, August 6, 2003.
Schwarzenegger ended weeks of
speculation about his candidacy for the
October 7 referendum. (Fred Prouser/
Reuters
Producer and director Bruce Paltrow has died
at the age of 58 in Rome, Italy, the U.S.
Consulate said on October 3, 2002. Paltrow
had suffered from throat cancer for several
years, but the cause of his death was not
immediately known. He is seen with his
daughter actress Gwyneth Paltrow after the
Academy Awards in Los Angles in March 21,
1999 file photo. (Fred Prouser/Reuters)
Names and Faces
President George W. Bush makes a
statement in the Rose Garden while
Secretary of Defense Donald Rumsfeld
looks on, July 23, 2003. Rumsfeld said the
United States would release graphic
photographs of the dead sons of Saddam
Hussein to prove they were killed by
American troops. Photo by Larry Downing/
Reuters
Intuition
George Bush
Names and Faces: Pipeline
 
Proper Name Extraction
 
Face Detection
 
Face Rectification & Representation
 
Assign names to faces by clustering
President George W. Bush makes a
statement in the Rose Garden while
Secretary of Defense Donald Rumsfeld
looks on, July 23, 2003. Rumsfeld said
the United States would release graphic
photographs of the dead sons of
Saddam Hussein to prove they were
killed by American troops. Photo by
Larry Downing/Reuters
Proper Name Extraction
President George W. Bush makes a
statement in the Rose Garden while
Secretary of Defense Donald Rumsfeld
looks on, July 23, 2003. Rumsfeld said
the United States would release graphic
photographs of the dead sons of
Saddam Hussein to prove they were
killed by American troops. Photo by
Larry Downing/Reuters
Detected Names:
President George W. Bush,
Defense Donald Rumsfeld,
Saddam Hussein.
Face Detection
President George W. Bush makes a
statement in the Rose Garden while
Secretary of Defense Donald Rumsfeld
looks on, July 23, 2003. Rumsfeld said
the United States would release graphic
photographs of the dead sons of
Saddam Hussein to prove they were
killed by American troops. Photo by
Larry Downing/Reuters
 
 
Face Detector (Mikolajczyk Ph.D
Thesis 2002)
45,000 large faces
 
mainly frontal
 
Scaled to 86x86 pixels
Rectify Faces
 
 
SVM feature detectors - trained to recognize nose,
corners of eyes and mouth
Determine affine transformation between detected
and canonical positions
 
 
Use gradient descent for fine alignment.
Project images into canonical pose
Face Representation
 
Faces initially represented as vectors of pixels
 
KPCA -- dimension reduction onto nonlinear basis.
 
 
1. Compute kernel function, K(i,j) for each pair of images
 
2. Calculate eigenvectors of kernel matrix and project
Nyström Approximation
 
Avoid computing all of K
mxm
Approximate C as:
NxN
 
Only compute N*m entries in K
 
Only compute eigenvectors of mxm matrices
Face Representation (cont.)
 
Linear Discriminants Analysis
 
Discriminants take advantage of class information
 
Don’t know classes
 
Build discriminants using one name/one face items,
project all face vectors into this space
Image Model
Learn what each individual looks like:
Face Model: P(Person i | Face Image)
Depiction Model
P(Pictured | Context)
Yes/No
multiple independent cues
What can context indicate about
depiction?
President George W. Bush
makes a statement in the Rose
Garden while Secretary of
Defense Donald Rumsfeld
looks on, July 23, 2003.
Rumsfeld said the United
States would release graphic
photographs of the dead sons
of Saddam Hussein to prove
they were killed by American
troops. Photo by Larry
Downing/Reuters
Depiction Model
P(Pictured | Context)
Yes/No
multiple independent cues
Cues: POS tags before and after name,
location in caption, distance to closest
. , ( ) (L) (C) (R) left right center
Cue Combination:
Maximum Entropy Model
President George W. Bush
makes a statement in the Rose
Garden while Secretary of
Defense Donald Rumsfeld
looks on, July 23, 2003.
Rumsfeld said the United
States would release graphic
photographs of the dead sons
of Saddam Hussein to prove
they were killed by American
troops. Photo by Larry
Downing/Reuters
Goal:
1.) Learn what each individual looks like -learn image model for each person.
2.) Learn what contextual cues indicate
depiction – learn depiction model across
names.
Achieve both of these by considering a big
assignment problem.
Assignment Problem
Assignment Problem
Leonardo DiCaprio
Anna Kournikova
Colin Powell
Denzel Washington
George Bush
Donald Rumsfeld
Jennifer Connelly
Britney Spears
Lucy Liu
Andy Roddick
Use clustering to find best assignments
Each distinct name in the dataset is a cluster
Assignment Problem
Leonardo DiCaprio
Anna Kournikova
Colin Powell
Denzel Washington
George Bush
Donald Rumsfeld
Jennifer Connelly
Britney Spears
Lucy Liu
Andy Roddick
Assign each face to one of the detected names
Assignment Problem
Leonardo DiCaprio
Anna Kournikova
Colin Powell
Denzel Washington
George Bush
Donald Rumsfeld
Jennifer Connelly
Britney Spears
Lucy Liu
Andy Roddick
Update Appearance Model for each name
and Language Model for all names
Assignment Problem
Leonardo DiCaprio
Anna Kournikova
Colin Powell
Denzel Washington
George Bush
Donald Rumsfeld
Jennifer Connelly
Britney Spears
Lucy Liu
Andy Roddick
Repeat until convergence
Results
Without – CEO Summit
With – Martha Stewart
Without – James Bond
With – Pierce Brosnan
Model
Without – Vice President Dick
Cheney
With – President George W. Bush
Accuracy of labeling
Vision model, No Lang model
67%
Vision model + Lang model
78%
Results - Clusters
Results - Depiction
Classifier
% correct
Baseline (all pictured)
67%
Learned Lang Model
86%
IN - pictured,
OUT - not pictured
Results
British director Sam Mendes and
his partner actress Kate Winslet
arrive at the London premiere of
'The Road to Perdition', September
18, 2002. The films stars Tom
Hanks as a Chicago hit man who
has a separate family life and costars Paul Newman and Jude Law.
REUTERS/Dan Chung
World number one Lleyton Hewitt of
Australia hits a return to Nicolas
Massu of Chile at the Japan Open
tennis championships in Tokyo
October 3, 2002. REUTERS/Eriko
Sugita
Results
US President George W. Bush (L) makes
remarks while Secretary of State Colin
Powell (R) listens before signing the US
Leadership Against HIV /AIDS ,
Tuberculosis and Malaria Act of 2003 at the
Department of State in Washington, DC.
The five-year plan is designed to help
prevent and treat AIDS, especially in more
than a dozen African and Caribbean nations
(AFP/Luke Frazza)
German supermodel Claudia Schiffer gave
birth to a baby boy by Caesarian section
January 30, 2003, her spokeswoman said. The
baby is the first child for both Schiffer, 32,
and her husband, British film producer
Matthew Vaughn, who was at her side for
the birth. Schiffer is seen on the German
television show 'Bet It...?!' ('Wetten Dass...?!')
in Braunschweig, on January 26, 2002.
(Alexandra Winkler/Reuters)
Face Dictionary
http://tamaraberg.com/faces/faceDict/NIPSdict/index.html