raagacom

Question Answering System
Based on Community QA
Xiaoyan Zhu
Tsinghua University, Beijing, China
[email protected]
Outline
• Introduction
• System overview
• Main work
– Similar question retrieval
– Answer generation
• Conclusion
Motivation
• Problems with general QA systems
– Restricted to certain well known simple types
question, such as factoid questions.
– Manually made databases
• Problems with Community QA portals
– 20% best answers are not the best
– 50% best answers are not unique -> redundant
– Users have to wait a long time for the answer
(average 14 hours from 100 thousands questions)
Our Idea
• Get information from cQA databases
to enhance the power of general QA
system, specially for complex
questions.
System Overview
Main Work
• Similar Question retrieval
– Question analysis to determine the type of the
question, and extract useful information
– Question retrieval to get candidate answers
• Answer generation
– Information distance based answer summarization
method
– Two answer summarization strategies
Similar Question Retrieval
• Question analysis
– NLP tool is used for POStagging to make word
segmentation and get POS of the words
– Keywords words extraction
• Noun, verb, long adjective, number and time and so on
• TF-IDF value is used to select most important words as
keywords
– Question type classification
• 6 types: for fact (e.g. who), for reason (e.g. why), for
solution (e.g. how), for definition (e.g. what is), for
resource (navigation), others.
• Method is based on SVM and rules
Similar Question Retrieval (con.)
• Similar question extraction
– Synonym expansion for keywords by some
dictionaries (Cilin in Chinese) & wiki concepts
– Questions matched with the most keywords or its
synonym expansions will be selected as similar
questions
Similar Question Retrieval (con.)
• Similar question reranking
– Rerank all similar questions selected by similarity
metrics
• Cosine similarity
• Max overlap
• Longest common substring
• Longest consecutive common substring
Workflow of Similar Question Retrieval
user’s
question
word
segmentation
and POSTagging
Keyword
extraction
cQA database
candidates
candidate
reranking
similar
questions
Answer Generation
• One question has several answers (3.4
averagely)
• Answer generation methods
– Select one best answer
– Summarize all answers to generate new one
• Strategy
– If all candidate answers are similar enough: select
a comprehensive one
– Otherwise make summary from all answers
Answer Generation (con.)
• Algorithms
– Information distance among many objects
• based on Kolmogorov complexity
min K ( x1...xn | xi )  Em ( x1...xn )  min  Dmax ( xi , xk )
i
i
k i
• The left–hand could be interpreted as the most
comprehensive object that contains the most
information about all of the others.
• The right-hand could be interpreted as the most typical
object that is similar to all of the others, which would
be used to select the sentences to make a summary.
Reference: CIKM 2008
Example: “Russian submarine Kursk explosion”
from Wikipedia
(original document: 1894 words, almost 900 sentences
13
Human sammary
Russian submarine Kursk explosion
• vital August 12,2000
• vital Norway provides rescue assistance.
• vital Britain provides rescue assistance.
• vital Norwegian divers confirm flooding entire sub.
• Vital Russian officials at first opine sinking resulted from
collision.
• okay Two US subs monitored exercise involving Kursk.
• okay US sub in area of sinking heard explosion.
• okay Russians accept likelihood of on-board explosion.
……
18 sentences are vital;
6 sentences are okay.
Summary ( 5 sentences)
•
【1】On the morning of 12 August 2000, as part of a naval exercise, ''Kursk''
was to fire two dummy torpedoes at a Kirov class guided missile cruiser,
''Russian cruiser Pyotr Velikiy'', the flagship of the Northern Fleet.
•
【2】The second explosion ripped a hole in the hull of the craft, which was
designed to withstand depths of 1,000 meters .
•
【3】British and Norwegian deep-sea divers reached the ninth compartment
escape hatch on Sunday, 20 August.
•
【4】Shortly afterwards, the Russian government investigation into the
accident officially concluded that a faulty torpedo sank Kursk in the Summer
of 2000.“
•
【5】In its 2002 review of two books on this topic, "''Kursk'', Russia's Lost
Pride" and "A Time to Die: The ''Kursk'' Disaster" The Guardian says: ''"The
hopelessly flawed rescue attempt, hampered by badly designed and decrepit
equipment, illustrated the fatal decline of Russia's military power.
Explosion
Explosion
Rescue process
Investigation of the cause of the accident
Conclusion
Evaluation results of the model in TACs
TAC 2008, Update Summarization
Evaluation Method
Average Modified Score
Macroaverage Modified Score
with 3 models
Average Linguistic Quality
Average Overall Responsiveness
Best
0.336
0.331
Our Result
0.309
0.304
Rank
5/58
5/58
3.333
2.667
2.958
2.667
3/58
1/58
TAC 2009, Summarization
Cluster
Evaluation Method
AVG Modified Score
MacroAVG Modified
AVG Linguistic Qualilty
AVG
Overall Resp.
2010/5/22
Traditional
Results
Rank
0.311
9
0.316
9
5.682
3
4.955
2
Update
Results
0.296
0.292
5.886
5.023
Rank
4/52
4/52
1/52
1/52 16
Examples: best answer selection
Original Query
How to improve my English?
Similar Questions
• How I improve my English?
• How can I improve my English?
• How can I improve my oral English?
• How can I improve my English fastly?
• How can I improve my spoken English?
• Guys, how can I improve my English?
Examples: best answer selection (con.)
• Answers from selected questions
Answer 1
Firstly read, read, read. It really helps a lot. English classes are okay too but
set
goals,2 like everyday read a chapter (or less) of a book and any words you
Answer
Finallook
Answer
don't
understand,
them
up.
Practiceskills
speaking
and
thinking inFind
English,
at who speak
The best
wayFirstly
to improve
your
language
is total
immersion.
people
read,
read,
read.
It
really
helps
a
lot.
English
classes
are
okay
but
first
it willand
be speak
difficultwith
andthem
sound
grammatical
incorrect
but
you'll
get
the hang
of buttoo
English
only
in
English.
You
may
find
this
difficult
at
first,
this
will
Answer
3 set goals, like everyday read a chapter (or less) of a book and any words you
itbesoon.
the
best method
for learning.
It seems
your
writingspeaking
skills are and
ok, so
most likely
you just
don't
understand,
them
up.
Practice
thinking
in English,
at
Go
totapes
more
public
places
(not alook
crack
house
lol)
Also
and
books
on
English
will
help
a
great
deal.
need
to
speak
the
language.
Also
I've
known
people
who've
learned
English
purely
will beudifficult
sound grammatical incorrect but you'll get the hang of
some
where
niceitwhere
should and
be xcepted
Answer
4 first
through movies.
it soon.
Just try reading
good English books or novels. When ur reading them read them loudly.
Also
tapesbooks.
and books
English
willmake
help up
a great
also
read5the grammar
And on
then
you can
smalldeal.
stories and recite them in
Answer
front
some
friends.
That will
increase
your knowledge
the language
and will also
Sorryofdear
theres
no better
option
than reading
books to about
improves
ones English.
make
youyou
fluent
it. newspapers,if you don't like story books.
Anyways
caninread
Answer
6
Or
you
can
tryyou
this...
Find
out
what
canItread
like comics,suspense,humor
etc. works on grammar and proper
Take
a
speech
class.
will help
you speak in public and also
http://www.1-language.com/Englishcourse/index.htm
Speak
to yourself
in English
andget
try alot
to answer
questions
to yourself.
annunciation.
I think
you would
more out
os a speech
class. Watch
Checkgood
it out.English
Good
luck
!
Answer
7
movies.
Try watching television -- that will help you begin to understand spoken English. Then try
talking
Answerto8people in your everyday life -- waiters/waitresses, bus drivers, store clerks,
anyone
-- most
try to be
helpful.
Start talking
on will
the phone
with
someone.
Examples: answer summary
Original Query
How to listen online musics?
Similar Questions
• How can I listen to music online?
• How can I listen music online?
• MP3 musics?
Examples: answer summary (con.)
••
••
••
••
••
••
••
••
••
••
••
••
••
••
Answer
Answer1:1:Windows
Windowsmedia
mediaplayer
playerwhen
whenyou
youhave
havethat
thatgo
gototoguide.
guide.
Answer
Answer2:2: [email protected],
[email protected],mtv.com,
mtv.com,vh1.com,
vh1.com,real.com
real.com
Answer
Answer3:3: Either
Eitheryahoo
yahoomusic
musicororhttp://www.mtv.com?
http://www.mtv.com?I Iguess
guessthose
thosetwo
twoare
aresafe...
safe...but
butititall
alldepends
dependson
on
ururconnection
connectionspeed...
speed...You‘ll
You‘llfreak
freakififuuhaf
hafaaslow
slowititconnection...well...Good
connection...well...Goodluck!
luck!
Answer
Answer4:4:Check
Checkout
outthe
thewebsite
websitefor
foryour
yourfav
favradio
radiostation.
station.Most
Mostradio
radiostations
stationsnow
nowallow
allowpeople
peopletotolisten
listen
online
online
Answer
Answer5:5:Well,
Well,http://www.raaga.com
http://www.raaga.comwill
willdefinitely
definitelywork
workfor
foru,u,its
itsthe
theno.
no.1....but
1....butthr
thrare
arealso
alsoaacouple
coupleofof
other
othersites
sitesthat
thatwill
willhelp
helpuulike
likehttp://www.raagalahari.com
http://www.raagalahari.comand
andhttp://www.andhravilas.com
http://www.andhravilas.com
Answer
Answer6:6:Go
Gototowww.musicindiaonline.com.
www.musicindiaonline.com.They
Theyhave
haveaavast
vastcollection
collectionofofsongs
songsininvarious
variousindian
indian
languages
languagesasaswell
wellasasEnglish.
English.The
Thedirections
directionsare
areeasy
easytotofollow
followand
andthe
thereception
receptionisisclear.
clear.
Answer
Answer7:7: http://www.blastro.com;
http://www.blastro.com;http://www.smashitsusa.com;
http://www.smashitsusa.com;http://www.rhapsody.com;
http://www.rhapsody.com;
http://www.live365.com
http://www.live365.comthis
thisone‘s
one‘sthe
thebest
besttotome
mecause
causeitithas
hasevery
everygenre
genreofofmusic
musicyou
youcan
canthink
thinkof.
of.Don’t
Don’t
sign
signup
upfor
forthe
thevip
vipcause
causeititisn‘t
isn‘tfree.
free.
Answer
Answer8:8: Try
Trynapster
napster
Answer
Answer9:9: Yahoo
Yahoolaunchcast
launchcast
Answer
Answer10:
10: I Iusually
usuallygo
gototomtv.com
mtv.comand
andtype
typeininthe
thename
nameon
onartist
artistsearch
searchand
andI Ifind
findthem
them
Answer
Answer11:
11: Download
Downloadwinamp
winampfrom
fromwinamp.com
winamp.comand
andstart
startitit, ,ininoption
optionthere
thereisismedia
medialibrary
librarygoto
goto
shoutcast
shoutcastradio
radiochoose
choosestation
stationgenre
genreand
andlisten
listenonline
onlinemusic.
music.
Answer
Answer12:
12: MP3s:
MP3s:free
freeand
andlegal.
legal.
Answer
Answer13:
13: There
Thereisisno
nosuch
suchthing
thingasasfree
freesongs
songstotodownload
downloadlegally!
legally!
Answer
Answer14:
14: Most
Most.mp3s
.mp3swill
willbe
bequite
quitelarge
largeininsize.
size.The
Thebetter
betterthe
thequality,
quality,the
thelonger
longerthe
thesong
songwill
willresult
resultinin
aalarge
largefile
filesize.
size.Most
Mostsongs/.mp3s
songs/.mp3s(that
(thatI Ihave)
have)are
arebetween
between1-5
1-5MB,
MB,which
whichwill
willtake
takequite
quiteaawhile
whiletoto
download
downloadon
ondial-up.
dial-up.
Additional Information
Recommendation
• The statistic information extracted from cQA
database could be used as additional
information recommended to a query log
which lacks something in detail.
Conclusion
• Community QA based answer generation is a
good solution for QA system with the complex
question such as why and how.
• The reuse of information collected from real
internet users could enhance the power of QA
system, which is more natural, appropriate for
the practical applications.
• Questions collected by cQA or query log could
be used for additional information
recommendation.
Our Goal
• Integrate as many as possible sources to make
a large scale open domain question and
answering system
– General search engines (e.g. Google), vertical
search engines, cQA databases (e.g. Yahoo!
Answers), Encyclopedias (e.g. Wikipedia); API of
special websites (e.g. translator, weather forecast,
news website,……)
How to cure
a bad cold?
How to say “I love
you” in Chinese?
Canon G9
Battery is good, poor display
Weather in
Beijing
Who is Obama
ONE agent
NO click
NO selection
Thank you !