AutoCWW Tutorial

Steps of the Cognitive Walkthrough for the Web (CWW)
Navigation System Analysis
Table of Contents
Part 1. Web-based interface . . . . . . . . . . . . . . . . . . . . . . . . . . 2
Part 1. Readability Evaluation Tool . . . . . . . . . . . . . . . . . . . . 3
Part 3. LSA measures and state-of-the-art parameters . . . . 3
Part 4. Formulate set of user goals . . . . . . . . . . . . . . . . . . . . 4
Part 5. Example of web-based CWW for real website. . . . . . 5
Part 6. Compute predicted mean total clicks for each user
goal, and test predictions in the lab . . . . . . . . . . . . . . . . . . .14
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .17
Appendix A: Elaborated Encarta headings and links . . . . .18
Appendix B: Sample user goals for Online Encyclopedia .29
2
Steps of the Cognitive Walkthrough for the Web (CWW)
Navigation System Analysis
1 Web-based interface
This tutorial assumes that the reader is familiar with Latent Semantic Analysis and with the
Cognitive Walkthrough for the Web (CWW). If not, the reader can read the papers listed in the
References section, track and download LSA papers at http://lsa.colorado.edu, and track and
download papers on CWW and ten experiments to date validating CWW via
http://AutoCWW.colorado.edu/~blackmon/. The server name, AutoCWW (see Figure 1)
indicates the long-range goal of the CWW research group to incrementally automate the CWW
procedures to make them more efficient for researchers, HCI students – and, ultimately,
collaborating web designers/developers, particularly those engaged in building informational
websites that offer medical/health and scientific information to the general public.
Figure 1 shows the tools that are available on the web-based interface as of July 2004. In the
Figure 1. Home page of http://AutoCWW.colorado.edu, showing tools available in July, 2004.
This home page appears immediately after login at http://AutoCWW.colorado.edu.
Created July, 2004
3
sections that follow, we will identify available tools by the name shown in Figure 1, but – a word
of caution – the web-based interface at http://AutoCWW.colorado.edu may be updated with
additional tools and help information before the next update of this tutorial.
2 Readability Evaluation Tool
The Readability Evaluation Tool is designed as a plug-in to Microsoft Word for Windows,
appearing under “Tools” on the Word menu. Both the Readability Evaluation Tool and the
accompanying manual for installing and using the tool can be downloaded at
http://AutoCWW.colorado.edu/Readability/ReadabilityTool.html.
3 LSA measures and state-of-the-art parameters
3.1 Similarity
The Latent Semantic Analysis (LSA) measure of similarity – the cosine of the angle between the
vectors of two texts – is the LSA measure that is best known and most widely used. CWW
employs the LSA cosine as a measure of the information scent between a given user goal text
and each heading text and link text on a particular webpage.
•
Weak scent is a goal-link cosine for the correct link with a cosine <0.10
•
Confusable heading/link texts are defined as a pair of texts with a cosine !0.60
The LSA cosine is also used by the Readability Evaluation Tool to assess appropriateness of the
content webpages for a particular target audience of the website, setting the following parameters
for the cosine values:
•
Sentence-to-sentence coherence: minimum mean cosine of 0.35
•
Key concepts in comparison with relevant text delineating the key concepts: !0.60
3.2 Familiarity
LSA offers two measures of familiarity: (1) frequency of a particular word within a semantic
space designed by scientifically sampling documents that have typically been read by a particular
user group, and (2) term vector length of a word or phrase within such a semantic space. Term
vector length measures background knowledge related to a topic (e.g., archaeology) for the group
of users represented by the semantic space. A user must know the meanings of the words in the
heading and link texts in order to perceive information scent when the headings/link texts are
highly similar to the user’s goal. Therefore, information scent is diminished if a heading/link text
consists exclusively, or even partially of low frequency words. Information scent is also reduced
if the percentage of low frequency words is too high in a particular subregion of the webpage,
making it impossible to infer the probable meaning of the low frequency words from the
surrounding context of familiar words. Here are the currently values of the relevant parameters:
•
Word frequency !15 in the selected semantic space is required for the word to be
considered sufficiently familiar, else the word is low frequency and unfamiliar
•
Percentage of low frequency words damages comprehension of a content article if above
8% low frequency words – measured by dividing the number of unique low frequency
Created July, 2004
4
words (words with a frequency of <15 in the semantic space) by the total number of
unique words in the text.
•
Term vector length >0.55 for 1-word link/heading, else !.80. Term vectors below these
one-word and multiple-word thresholds are considered to imply insufficient background
knowledge of the topic.
3.3 Similarity and familiarity simultaneously measured
To simulate text elaboration during comprehension we assume that the words that are activated
in the user’s memory to elaborate the printed text are words that are both highly familiar and
highly similar to the printed text being read by the user.
•
“Highly familiar” is measured using a minimum word frequency of 50 in the semantic
space
•
“Highly similar” to the text being read is measured by using a minimum cosine of 0.50.
4 Formulate set of user goals
To design or re-design an actual website, the ideal is to interview prospective users of the
website to discover their goals for using the website. For example, the Osteosarcoma Online
website uses interviews of, and feedback from, osteosarcoma patients, members of patients’
families, and friends of osteosarcoma patients. The goal is to discover the information that these
users want to find on the website and to make sure the information is provided at the right level
of detail and level of general reading knowledge and background knowledge of medicine.
4.1 List user groups and semantic space that most validly represents
each user group
If experiments are to be run using American undergraduate students as experimental participants,
then the most appropriate semantic space is general reading knowledge of first-year college
(tasaALL). Other general reading knowledge semantic spaces are available for English-speaking
Americans at 3rd, 6th-, 9th-, and 12th-grade reading level. There is also a semantic space for
college-educated French language speakers, based on a corpus of a full year of Le Monde
newspaper. Additional semantic spaces are in preparation but not yet available.
4.2 Representative set of user tasks for each user group
So far, CWW has been applied to tasks where the user is attempting to find information on a
website, such as an online encyclopedia, a medical/health website for patients and other lay
users, or a website that provides scientific information.
4.3 Discount list of user tasks: full text or summaries of all content
webpages
The user goals that have been used in experiments by members of the CWW research team are
generally summaries of content articles available on the existing website being evaluated, using
these criteria:
•
Each summary presented to experimental participants is typically 100-200 words.
Created July, 2004
5
•
In order to ensure that the summary accurately represents the full article, there must be a
cosine of 0.80 or higher between the summary and the full article text.
Microsoft Word contains a summarization tool that produces a first draft of the summary in
complete sentences. The draft sometimes needs to be hand edited for coherence or to increase the
cosine between the summary and full text of the article to the threshold value of 0.80 or higher.
5 Example of web-based CWW for real website
This example is designed to make predictions about user performance on one webpage,
distinguishing which tasks will be easy to do and which tasks will encounter significant usability
problems. The example webpage, shown in Figure 2, is the home page of Encarta Online
Encyclopedia (http://encarta.msn.com). These predictions have evolved from laboratory
experiments that use simulations of this webpage with college students. Extrapolating from
accumulated data for simulations of this webpage, we can predict the approximate mean number
of clicks (to an estimated accuracy of ± 2 clicks) that college students require to complete the
task. The measure “Mean Total Clicks” is a measure of problem severity that turns out to be
closely correlated with another measure of problem severity, namely the approximate rate of task
failure, i.e., percentage of users who will not be able to complete the task in 130 seconds or less.
To keep this example as simple as possible we will also abide be the following constraints:
Figure 2. Encarta online encyclopedia showing nine categories that house a total of 93 links
Created July, 2004
6
•
Restrict attention to links and headings in the content area of the webpage
•
Group in a simple matrix, ignoring effects of graphic design elements on user behavior
•
Postpone detailed descriptions of CWW repair methods and their success rates.
5.1 Identify visuospatial-semantic groupings
The simplest approach applies CWW just to the headings and links in the content area of a single
webpage, allowing us to make and test predictions for tasks that can be accomplished with a
minimum single click on one webpage. For this example we will use CWW to predict human
performance finding encyclopedia articles on specific topics on the Encarta Online
Encyclopedia, a webpage that offers 93 links nested under nine different categories/headings.
Figure 2 shows the webpage as it first appears to users. Clicking on any of the nine categories
reveals the links nested under that one category heading, and white space separates groupings.
Figure 3 shows the webpage used in laboratory simulations of this experiment, displaying all 93
links in a simple three-by-three matrix with nine cells, one for each heading. Comparing Figures
1 and 2, we see that the actual webpage displays a few links that do not appear in the simulated
version. The top navigation bar in Figure 2 presents links to Home (home of the Microsoft
reference center), Encyclopedia (present location, the main page of a major subsite of the
reference center), Atlas, Dictionary, Multimedia, Magazines, Homework, Math Help, and Search
Encarta. None of these additional links, however, are relevant to the task of browsing the online
Figure 3. The task Find encyclopedia article about Hmong in the simulation of the Encarta
online encyclopedia with 93 total links nested within a matrix of nine headings
Created July, 2004
7
encyclopedia to find an article on a particular subject. (In order to focus exclusively on finding
articles by browsing we exclude the site search engine.) Therefore, the simplified matrix
presented in Figure 3 captures the essence of search tasks on the encyclopedia. College students
who participated in a CWW experiment did a series of pages that all looked like Figure 3 except
for variations in the orange box telling participants what to search for. The Figure 3 task is “Find
an encyclopedia article about Hmong.” Other tasks searched for such articles as Music Therapy,
Cybernetics, and Ferreting. The links nested under each of the nine categories are listed in
exactly the same order that they appear on the webpage. The visuospatial grouping in Figure 3 is
a matrix design with borders outlining each matrix cell to clearly distinguish each cell from other
cells. Headings stand out in contrasting color font, larger size font, and boldface for emphasis.
5.2 Identify insufficient familiarity problems
To accurately predict human performance on webpages one necessary step is to identify heading
and link texts that will probably be unfamiliar for a target group of website users. For
experiments, the goal texts should also be analyzed to ensure that they do not contain a lot of
Figure 4. Encarta heading and link texts ready to submit to Low Frequency Words Analysis in
the semantic space for third-grade general reading knowledge
Created July, 2004
8
unfamiliar words and concepts and, thus, are not difficult for experimental participants to
comprehend. There are two LSA-based measures of familiarity. The first is identification of all
the low frequency words and computation of the percentage of low frequency words. The second
is finding link texts with low term vector length.
The Readability Evaluation Tool (plug-in to Microsoft Word for Windows) provides the most
efficient way to identify low frequency words, compute the percentage of low frequency words,
and edit to reduce the percentage. Alternatively, use the web-based tool by clicking the link
“Low Frequency Words Analysis” on the home page shown in Figure 1. Figure 4 gives a
snapshot of the text submitted to Low Frequency Words Analysis as well. To find exact
frequencies of words, submit the words to the “Frequency Analysis” tool. Because the Frequency
Analysis tool reports the frequency of each unique word only once, it is possible, but timeconsuming and tedious, to use the “Frequency Analysis” tool to compute word frequency for
each and every word in a text, sort the list by word frequency, and then compute the percentage
of low frequency words by dividing the unique number of low frequency words by the total
number of unique words in the passage.
By whatever method we choose to do the computation, we can discover that there are 154 unique
words in the nine headings and 93 links shown in Figure 3. Of these 154 words, seven (5%) are
low frequency words even in the first-year college semantic space (e.g., Australasia,
paleontology, archaeology, monerans, and occult). Low frequency words cause avoidable
Figure 5. How third graders see the task “Find encyclopedia article about Hmong,” showing
low frequency words in red font and low term vector heading/link texts in bold italics
Created July, 2004
9
usability problems even at the 5% level, but users with third-grade reading level face 51% low
frequency words (79/154) on the same page, an overwhelming source of usability problems. For
users with sixth-grade reading level, 18% of the words are low frequency (27/154).
On webpages, users tend to ignore links that are unfamiliar, because the correct link has little or
no information scent for a user who has rarely or ever encountered the words in the link. Figure
5 shows all the usability problems that would face a third grader trying to find an article in the
Encarta encyclopedia. Red font indicates words that have low frequency in the third-grade
semantic space, and bold italic font indicates headings and links that have low term vector
length. Four out of nine headings (44%) have low term vector length, and 51 out of 93 links
(55%) have low term vector length. Third-graders could probably find the familiar links “Fish”
or “Birds” or “Plants” under the familiar heading “Life Science,” but third-graders might never
find the familiar link “Music” under the unfamiliar heading “Performing Arts” or click any link
under the unfamiliar heading “Religion & Philosophy.”
A high percentage of low frequency words makes it difficult to infer or guess the meaning of an
unfamiliar word from the surrounding context, because the surrounding context is also full of
unfamiliar words. Even when a link text does not use any low frequency words, however, the
user may still not have sufficient background knowledge of the topic. The term vector length
measures the degree to which the semantic space contains information closely related to a
heading or link text. In the 10 experiments run to date by CWW researchers, there has been a
consistent result that identifies low background knowledge – as measured by term vector length
– as a serious impediment for users searching for information that requires clicking an unfamiliar
link in order to find it.
Table 1 displays the output from the AutoCWW Unfamiliar headings/links analysis for the 93
links on the Encarta.msn.com website. This analysis identifies as unfamiliar any link that is
Table 1. Unfamiliar links in college-level semantic space for Encarta Online Encyclopedia (low
frequency words in red font, word frequencies and one-word term vector lengths in parentheses)
One-word links
Term vector
length !0.55
Links with two or more words
Term vector
length <0.80
Paleontology (4)
0.06
Legends (96) & Folklore (21)
0.39 (0.32, 0.11)
The Occult (11)
0.08
Anatomy (76) & Physiology (48) 0.51 (0.30, 0.24)
Scripture (16)
0.10
Theology (31) & Practices (456)
0.76 (0.20, 0.67)
Archaeology (11)
0.10
Calendar (143), Holidays (118),
& Festivals (43)
0.77 (0.49, 0.28,
0.23)
Mythology (38)
0.17
Anthropology (65)
0.50
Pets (199)
0.51
Communications (213) 0.55
<0.80, but a better parameter for one-word links appears to be "0.55. Term vector lengths for
two or more words are always longer than the term vector length for single words, so it is more
Created July, 2004
10
reasonable to have separate parameters for one-word heading/link texts and for heading/link texts
of two or more words.
There is no quick fix for lack of background knowledge. Repairing unfamiliar links can be done
by altering the heading/link text, e.g., changing the link “Paleontology” to “Paleontology and
Fossils,” or changing the link “Anthropology” to “Anthropology and Cultures.” Such changes
can help when the link text uses low frequency words, but they cannot genuinely compensate for
insufficient background knowledge. Another repair provides an alternative path to the
information by clicking a familiar link that is highly similar link to the user's goal.
Figure 6. Set of raw link texts ready to submit for elaboration with a minimum frequency of 50
and minimum cosine of 0.50.
Created July, 2004
11
Table 2. Selected raw link texts and their elaboration by AutoCWW “Elaborate Links”
5.3 Elaborate link and heading texts to simulate reading
comprehension
The reading comprehension process elaborates the text that appears on the page by activating
words that are semantically similar to the text being read (Kintsch, 1998). Activation of
semantically similar words also accounts for the rapidity with which people learn new
vocabulary, because learning or improving learning of one new word simultaneously adds to the
knowledge of semantically related words (Landauer & Dumais, 1997).
Selecting the link “Elaborate Links” on the AutoCWW home page (Figure 1) makes it possible
to submit a whole set of raw link/heading texts for LSA Near Neighbors analysis and to set the
parameters. Figure 6 shows a set of raw text links ready to be submitted. Since LSA Near
Neighbor analyses are time-consuming, it helps to not submit more than 50 raw texts at one time
(to avoid browser timeouts). The semantically related words most likely to be activated are
words that are both highly familiar and semantically close, so we recommend setting the
minimum word frequency at 50 and the minimum cosine at 0.50, as illustrated in Figure 6.
Well-designed raw link texts use a minimum number of words and occupy a minimum amount
of screen space but use words that are rich in meaning, activating many other semantically
related, familiar words. Table 2 displays a sample of raw text links and the elaborated texts
output from the “Elaborate Links” tool. As Table 2 demonstrates, even a single word can activate
a host of related words in the mind of the user reading a one- or two-word link.
To capture the complex meaning of a heading and the cluster of links nested under the heading,
our current practice is to elaborate each heading text and add to it the elaborated texts of all the
subordinate links, creating a large cluster of words. For example, the heading “Performing Arts”
and the elaborated versions of its six subordinate links – “Music,” “Musicians & Composers,”
“Musical Instruments,” “Dance,” “Theater,” and “Cinema, Television, and Broadcasting” –
becomes the lengthy elaborated heading text below:
Performing Arts arts art artistic artists painters sculpture sculptures architecture
Theater theater actors actor playwright playwrights drama plays theatre comedy shakespeare costumes
performers broadway audiences scenery macbeth tragedy scenes theaters performances stage staged
elizabethan script theatres scene audience costume performer performed movie chorus
Musicians & Composers composers musicians jazz music singers beethoven musician orchestra musical
melody piano tunes symphony concert songs blues opera chord dance song rhythmic sing rhythm concerts
dancing singing tune singer violin dancer guitar rhythms performances danced instruments sung dancers
ballet sang tones instrument band drums chorus dances chant folk choir instrumental strings
Cinema, Television, & Broadcasting television viewers broadcast broadcasting tv commercials radio
stations entertainment media movies newspapers network news magazines advertisers networks radios
communications studio appearances channels
Music music jazz singers musicians songs composers tunes musical melody piano orchestra beethoven
musician concert song blues symphony sing opera rhythm singing tune dance rhythmic singer concerts guitar
sang dancing rhythms drums sung danced chord violin choir band chant dancer performances dancers ballet
instruments dances folk chorus flute tones strings instrument harp lessons sings
Dance dance dancing ballet dancer dancers dances music danced jazz musicians piano composers concert
orchestra singers musician opera tunes concerts performances beethoven songs melody blues musical
performers sing symphony costume
Musical Instruments instruments musical instrument music jazz melody piano musicians musician
guitar strings violin orchestra songs concert rhythmic tune rhythm blues
Raw Encarta link texts
Elaborated text, minimum frequency 50, minimum cosine 0.50
Created July, 2004
12
Artists
artists art painters artist paintings painting painter artistic sculpture
portraits sculptures painted paint portrait architecture museum
museums realism renaissance decorative arts sculptor patrons paints
gallery statues florence claude geometric style
Theater
theater actors actor playwright playwrights drama plays theatre
comedy shakespeare costumes performers broadway audiences
scenery macbeth tragedy scenes theaters performances stage staged
elizabethan script theatres scene audience costume performer
performed movie chorus
Music
music jazz singers musicians songs composers tunes musical melody
piano orchestra beethoven musician concert song blues symphony
sing opera rhythm singing tune dance rhythmic singer concerts guitar
sang dancing rhythms drums sung danced chord violin choir band
chant dancer performances dancers ballet instruments dances folk
chorus flute tones strings instrument harp lessons sings
Language
language spoken dialect languages dialects speak grammatical
vocabulary grammar expressive speaking slang english
Birds
birds bird feathers beaks beak wings eagle nest nests loon fly nesting
hummingbirds wing robins geese hawk flew pigeons feather gull
crows eagles cranes ostrich swan hummingbird pigeon owls fluttered
flying
Medicine
medicine medicines doctor doctors prescription sick medical clinic
Physics
physics physicists science sciences physicist biology geology
chemistry einstein scientific mathematics
Anthropology
anthropology anthropologist sociology anthropologists disciplines
humankind sociologists sciences mead societies human beings culture
scientifically psychology cultures
Scripture
sin, theology
Countries
countries underdeveloped industrialized nations
5.4 Identify goal-specific competing headings and links
The full set of elaborated heading and link texts for the Encarta encyclopedia is shown in
Appendix A, ready to copy and paste into AutoCWW “One-to-Many Analysis” for comparison
with each specific user goal. Sample user goals for Encarta Online Encyclopedia are shown in
Appendix B.
Created July, 2004
13
Figure 7 shows the One-to-Many webpage of AutoCWW with one goal – “Find an encyclopedia
article about Hmong” – in the upper box and a large set of nine elaborated heading texts and 93
elaborated link texts ready to submit for comparison with the goal. As specified on the website,
an empty line separates each elaborated text from the adjacent texts above and below it. One-toMany returns a two-column table of results with rows for each heading and link text. Column
one lists each one of the texts, and column two displays the cosine between each text and the
goal statement.
The next step is to copy that table of results and paste it into an Excel spreadsheet. Our practice
is to copy the results into the A and B column of an Excel sheet, specify whether the text is a
Figure 7. AutoCWW One-to-Many Comparison webpage forms ready to submit for comparison
of the Find Hmong goal with each Encarta heading and link text, using the college-level space
Created July, 2004
14
heading or a link text in column C, and then indicate the specific parent heading for each link in
column D (e.g., the parent heading “Performing Arts” for the link “Music”). That four-column
table is then copied and pasted in columns F-G-H-I, and then sorted first by heading or link and
then by descending cosine value. The “correct” heading(s) and link(s) are highlighted in green.
Goal-specific competing headings are headings with cosine values equal to or greater than the
cosine of the “correct” heading. Goal-specific competing links are links with cosine values equal
to or greater than 80% of the value of the cosine for the “correct” link. Both competing headings
and competing links are highlighted in yellow.
The underlying CoLiDeS cognitive model (Kitajima, Blackmon, & Polson, 2000) predicts that
users first focus their attention on the subregion that is most similar in meaning and then select
the link in that subregion that is most similar in meaning. Thus, the descending values of goalheading cosines predict which heading(s) users are apt to focus on, and the descending values of
goal-link cosines predict the links that users are most likely to click to find the target article.
Repairing these problems makes it possible for people to find items where they are most likely to
look, connecting multiple links to the target whenever required.
6 Compute predicted mean total clicks for each user goal,
and test predictions in the lab
This step begins with the output of the One-to-Many comparison and uses a formula for
predicting the mean total clicks that users will make on the Encarta webpage. The formula is
based on a multiple regression analysis of the data from three experiments.
The complete multiple regression-based formula computes the predicted mean total clicks =
+ 1.919
+ 1.420 times the number of competing headings
+ 0.281 times the number of competing links
+ 0.958 if correct link is unfamiliar
+ 1.231 if correct link has weak scent
The minimum path is a single click on the correct link that leads to accomplishing the goal, and
mean predicted total clicks for a task with no usability problems is 1.919 clicks.
Figure 8 shows the sorted results of the One-to-Many comparison for the goal “Find an
encyclopedia article about Fern.” This is a task that has no predicted usability problems. The
correct heading is “Life Science,” and “Life Science” is the heading predicted to be the heading
that users will focus on. The correct link is “Plants,” and the prediction is that “Plants will
almost always be clicked first. The link “Plants” is not unfamiliar, and it is not a weak-scent link
(goal-link cosine <0.10). Therefore, 1.919 is the predicted mean total clicks for Find Fern.
Actual data from the lab show that these predictions are very accurate for undergraduate
performance on this task:
•
1.1 mean total clicks on links for a minimum one-click path (vs. 1.919 predicted)
•
97% first clicked a link under Life Science, and 90% clicked Plants
•
100% success rate – everyone completed task in 1 or 2 clicks
•
9 seconds = mean solution time
Created July, 2004
15
Figure 8. Sorted results of One-to-Many comparison for the goal “Find encyclopedia article
about Fern,” with the “correct” heading and “correct” link highlighted in green.
A stunning contrast is provided by the task “Find an encyclopedia article about Hmong.” Figure
9 shows the results of the One-to-Many comparisons for the Hmong task. The “correct” heading
is Social Science, but the cosines for three headings are all higher than the cosine for Social
Science, making these headings goal-specific competing headings. Competing headings are
serious problems, pulling the user’s attention away from the “correct” heading. The formula adds
1.42 clicks for each of the three competing headings. The “correct” link, “Anthropology” has
two additional usability problems of its own. First, it is an unfamiliar link, and that adds 0.958 to
the predicted mean total clicks. In addition, it has weak scent (goal-link cosine <0.10), which
adds 1.231 to the predicted mean total clicks. Finally, there are competing links under the
correct and competing headings. Under the heading “History,” the highest cosine is 0.37 for the
link “People in United States History,” but there are two additional competing links that exceed
80% of that 0.37 cosine, “United States History,” and “History of Asia and Australasia.” Under
the heading “Art, Language and Literature” the link “Language” has the highest cosine and no
competitors. Finally, under the heading “Geography,” the highest cosine is for the link “U.S.
States, Territories, & Regions.” If we decide that real users will actually select that link then
there is only one competing link under “Geography.” If we rule out the highest-cosine link as a
likely choice (false positive), we instead choose two competing links under “Geography,” the
link “Regions of the World” and the link “Countries.” To be conservative we can do the
computation of mean total clicks using just one competing link under the “Geography” heading,
and that makes a total of five competing links.
Applying the formula to the Find Hmong goal, we get + 1.919 + 3 (1.42) for competing headings
+ 5 (0.281) for competing links + 0.958 for an unfamiliar correct link + 1.231 for a weak-scent
link. The total is 9.773, correlated with a predicted task failure rate of approximately 75%. The
actual human performance for 38 undergraduates who searched for Hmong in the unrepaired
condition of the experiment fell very close to predictions:
Created July, 2004
16
•
9.0 mean clicks (vs. 9.773 predicted)
•
74% task failure rate (inability to find the Hmong article within a time limit of 130
seconds vs. predicted 75% task failure rate)
•
Mean solution time of 124 seconds (close to the time limit of 130 seconds)
•
First-click success rate of only 3%
This predicted problem was, in fact, a serious problem for most of the people who attempted to
do it in the experiment.
Repairing goal-specific competing headings and goal-specific competing links requires making it
possible for people to find the target item by clicking any of the links that CWW analyses predict
are most likely for real people to click. For the “Find encyclopedia article about Hmong” tasks,
that means making it possible to reach the Hmong article by clicking any of the three most likely
links under the “History” heading, the Language link under the “Art, Language and Literature”
heading, and the “Regions of the World” and “Countries” links under the heading “Geography.”
Making these repairs produced significantly better performance (p <.0001) for the 38
undergraduates who searched for Hmong on the repaired webpage compared to the 38
Figure 9. Sorted results of One-to-Many comparison for the goal “Find encyclopedia article
about Hmong,” with the “correct” heading and “correct” link highlighted in green.
Created July, 2004
17
undergraduates who searched for Hmong on the original, unrepaired webpage:
•
2.1 mean total clicks (vs. 1.919 predicted mean clicks after all the problems had been
repaired)
•
0% task failure rate (all 38 experimental participants in the repaired condition found the
target article before the 130-second time limit expired)
•
Mean solution time of 41 seconds
•
First-click success rate of 43%
The Find Fern and Find Hmong tasks are especially close to predicted values, but overall the
CWW predictions are very accurate. The hit rate for predicted problems approaches 100% (0%
false alarms), the success rate for repairs (requires statistically significant improvement in
performance when repaired compared to the original predicted problem) ranges from 85% to
100%, and preliminary data on predicted no-problems finds correct rejections of at least 63%
with no serious problems among the misses.
7 References
Blackmon, M. H. (to appear 2004). Cognitive Walkthrough. In W. S. Bainbridge (Ed.), Encyclopedia of
Human-Computer Interaction, 2 volumes. Berkshire.
Blackmon, M. H., Kitajima, M., & Polson, P.G. (2003) Repairing usability problems identified by the
Cognitive Walkthrough for the Web. Chi Letters, 5: Proceedings of CHI 2003, 497–504 (ACM
Press).
Blackmon, M. H., Polson, P. G., Kitajima, M., & Lewis, C. (2002). Cognitive walkthrough for the Web.
Chi Letters, 4: Proceedings of CHI 2002, 463–470 (ACM Press).
Kintsch, W. (1998) Comprehension: A paradigm for cognition. Cambridge, U.K. & New York:
Cambridge University Press.
Kitajima, M., Blackmon, M. H., & Polson, P. G. (2000). A Comprehension-based model of Web
navigation and its application to Web usability analysis. In S. McDonald, Y. Waern & G. Cockton
(Eds.), People and Computers XIV—Usability or Else! (Proceedings of HCI 2000, University of
Sunderland, St. Peter’s Campus, UK, September 5–8, pp. 357–373). Springer.
Landauer, T. K. (1998). Learning and representing verbal meaning: Latent Semantic Analysis theory.
Current Directions in Psychological Science, 7, 161–164.
Landauer, T. K. (2002). On the computational basis of learning and cognition: Arguments from LSA. In
N. Ross (Ed.), The psychology of learning and motivation, 41, 43–84.
Landauer, T. K., & Dumais, S. T. (1997). A solution to Plato’s problem: The Latent Semantic Analysis
theory of acquisition, induction and representation of knowledge. Psychological Review, 104, 211–
240.
Landauer, T. K., Foltz, P., & Laham, D. (1998). An introduction to Latent Semantic Analysis. Discourse
Processes, 25, 259–284.
Landauer, T. K, Laham, D, & Foltz, P. W., (2000). Intelligent Essay Assessor. IEEE Intelligent Systems,
15(5) 27-31.
Created July, 2004
18
8 Appendix A: Elaborated Encarta headings and links
Art, Language, & Literature language art artistic artists arts artist dialect languages
dialects spoken painter painting painters paintings sculpture
National & Regional Literature national literature regional
Literature & Writing writing writers prewriting essays revising essay writer write draft
narrative readers expository writes assignments autobiography literature descriptive humorous
literary diaries diary compositions novelist
Architecture architecture sculpture art artists artistic statues sculptures sculptor paintings
painters architects renaissance style architect gothic designs painter painting revival artist
portraits styles decoration arts decorative
Artists artists art painters artist paintings painting painter artistic sculpture portraits sculptures
painted paint portrait architecture museum museums realism renaissance decorative arts sculptor
patrons paints gallery statues florence claude geometric style
Language language spoken dialect languages dialects speak grammatical vocabulary grammar
expressive speaking slang english
Writers & Poets poets poetry poet poems verse prose poem poetic whitman lyric poe writers
rhyme literature epic literary critic novelist lowell bryant imagery dickinson themes imaginative
figurative ballads romanticism novels romantic ironic metaphor hopkins rhythms joyce essays
edgar simile writer
Decorative Arts arts art artistic artists decorative painters artist painting sculpture paintings
painter sculptures architecture designs style
Legends & Folklore legends tales storytellers stories storyteller tale heroes myths story
National & Regional Art art artists artist paintings artistic painting painters painter sculpture
sculptures museum national museums portraits architecture painted
Painting, Drawing, & Graphic Arts sketches drawing drawings artist sketch sketching
pictorial drafting shading draw painting drafters artists art drafter lettering painters painter
graphic artistic geometric paintings portrait portraits drawn dimensions illustrations designer
representations blueprint
Sculpture sculpture sculptures art artists paintings artistic artist painting painters painter
architecture statues painted museum portraits museums statue portrait paint renaissance arts
Periods & Styles periods period styles
Photography photography photographers photographer camera film cameras shutter
photographs aperture photographic photographed films exposure images pictures photograph
studio
Life Science science sciences biology scientific geology physics life biologist physicists
Plants plants plant flowering mosses
People in Life Science science sciences biology scientific life geology physics
Medicine medicine medicines doctor doctors prescription sick medical clinic
Invertebrate Animals animals animal zoo zoos lions jellyfish backbones elephants
veterinarian
Fish fish trout tuna salmon fins fishing aquarium bait swims hook shark bass sharks fishermen
shellfish catch fisherman swim arf perch gills dolphin nets seafood fishes swam carp
Created July, 2004
19
Algae & Fungi algae chlorophyll alga fungi protist protists kelp spore multicellular fungus
yeasts protozoans hyphae molds seaweed mushrooms spores chloroplasts microscopic mold
protozoa mushroom phyla pigments rotting ponds parasites green decaying paramecium
decompose
Agriculture, Foodstuffs, & Livestock agriculture agricultural livestock farmers crop crops
wheat subsistence farming cultivation harvest harvesting soybeans farm hogs yields corn farmer
farms grain plows cultivate peanuts surpluses cultivated oats harvested barley export fertilizers
Mammals mammals mammal whales reptiles rodents reptile porpoises
Reptiles & Amphibians reptiles amphibians crocodiles lizards salamanders gills toads turtles
tadpoles frogs lizard snakes vertebrates scales tails fins toad hatch frog fishes eggs
Biological Principles & Concepts concepts principles concept theoretical
Anatomy & Physiology anatomy physiology
Environment environment environments environmental adapted adapt ecology adaptation
surroundings ecologists
Birds birds bird feathers beaks beak wings eagle nest nests loon fly nesting hummingbirds wing
robins geese hawk flew pigeons feather gull crows eagles cranes ostrich swan hummingbird
pigeon owls fluttered flying
Viruses, Monerans, & Protists viruses bacterium bacteria virus protozoa bacterial
protozoans mumps polio viral microbes host microorganisms pathogens protists colds infectious
protist infections infected rabies pneumonia measles influenza microscopic vaccine immune
pasteur immunity infection antibiotics disease villain tuberculosis
History history historians historian historical
History of Asia & Australasia asia asian malaysia strait afghanistan southeast
People in European History history european
People in United States History united states
United States History united states
African History african africans africa nigeria ghana congo tanzania niger ethiopia kenya
guinea history sudan mali sahara tribal
World History & Concepts world history
Ancient History history ancient historians civilization monuments antiquity civilized
History of the Americas history historians
European History history european europeans
Geography geography geographers boundaries
World Cities, Towns, & Villages cities towns suburbs villages metropolitan urban
Regions of the World world regions
Rivers, Lakes, & Waterways rivers lakes streams waterways dumped sewage polluted
dump phosphates dumping reservoirs tributaries erie
Parks & Monuments parks park
Countries countries underdeveloped industrialized nations
Canadian Provinces & Cities cities suburbs metropolitan provinces canadian urban
canadians quebec slums canada
Created July, 2004
20
Islands islands island hawaiian coral hawaii mainland philippines honolulu rico puerto
caribbean indies zealand ricans indonesia
Mountain Ranges, Peaks, & Landforms mountain peaks mountains ranges rugged
plateaus slopes sierra valleys mt peak everest foothills rocky ridges plateau climbers highest
appalachians rockies cascade mount steep summit towering himalayas alps landforms nevada
hills andes range
U.S. Cities, Towns, & Villages cities towns suburbs villages metropolitan rural urban
Maps & Mapmaking maps map mapmakers topographic globes mapping symbols contour
atlas symbol mapped elevation
Oceans & Seas oceans ocean seas salinity marine deepest
Exploration & Explorers explorers explorer explored exploration voyages exploring
columbus voyage 1492 sailed explorations magellan cabot expeditions christopher explore
portugal cartier leif claimed expedition isabella riches ferdinand newfoundland adventurers da
U.S. States, Territories, & Regions states united regions
Religion & Philosophy religion religions teachings religious jesus faith christianity christ
christians christian philosophy testament beliefs salvation belief god preached worship
mohammed followers sin islam muslims spiritual holy prayer muhammad prayers divine pray
eternal bible protestants catholic churches
Theology & Practices practices theology religious
Mythology mythology
Religious Figures religious religion worship christ salvation churches religions church
puritans theology protestants saints puritan bible jesus catholic prayer beliefs protestant catholics
faith teachings ritual preaching reformation christian sin testament christianity christians
preached spiritual congregation preach persecution god hutchinson judaism sabbath orthodox
holy prayers rituals pray sins
Philosophy philosophy philosophical metaphysics plato philosophers philosopher aristotle
socrates thinkers principles disciplines contemplation theology
Religions & Religious Groups groups religious religion religions group protestants christ
christians jesus protestant beliefs catholics teachings christianity christian worship
Scripture sin theology
The Occult
Physical Science & Technology science physical sciences physics scientific biology
geology technology
Construction & Engineering construction architectural engineering engineer exterior
architect constructed specifications architects masonry engineers plumbing design building
foundation construct tile projects designers builders
Chemistry chemistry chemist chemists biology physics science sciences chemical
Earth Science earth science geology scientific sciences
Computer Science & Electronics computer computers microcomputer software
computerized computing video graphics electronic diskette capability science calculations
electronics
Machines & Tools machines machine tools wedge tool levers fulcrum lever portable
Created July, 2004
21
People in Physical Science physical science sciences physics scientific
Astronomy & Space Science space science astronomy shuttle sciences physicists physics
biology scientific astronaut astronauts
Paleontology
Industry, Mining, & Fuels coal industry mining mines industries mined fuels mills
industrial petroleum
Physics physics physicists science sciences physicist biology geology chemistry einstein
scientific mathematics
Transportation transportation railroads freight rail highways railroad
Communications communications networks communication telegraph broadcasting
telephone communicating sender stations messages message network receiver radio television
broadcast media
Mathematics mathematics mathematical physics algebra science math
Military Technology military technology technologies
Time, Weights, & Measures measures weights measure measuring measurement metric
measurements measured weight kilogram grams si weighing meter
Military military armed defense strategic generals civilian
Social Science social sciences science sociology sociologists scientific anthropology biology
Economics & Business business businesses proprietorship entrepreneurs profit
Organizations organizations organization organizational membership organized association
associations
Institutions institutions institutional institution ideology
Political Science political science politics sciences scientific elites
Psychology psychology psychologists psychologist sociology behavior anthropology
scientifically psychological
Law law laws enforced legal enforcement enforce tort statute wrongs
Education education educational schools schooling vocational educators educating colleges
curriculum certification educate universities guidance elementary enrolled training
Anthropology anthropology anthropologist sociology anthropologists disciplines humankind
sociologists sciences mead societies human beings culture scientifically psychology cultures
Military military armed defense strategic generals civilian
Sociology & Social Reform social sociology sociologists sociological sociologist needy
Calendar, Holidays, & Festivals calendar holidays holiday celebrate festival celebrated
celebration
Archaeology prehistoric anthropologists archaeologists anthropology anthropologist
Sports, Hobbies, & Pets sports athletes athlete athletic athletics sport coaches games teams
basketball football hockey olympic team soccer championship tennis boxing players
Sports sports athletes athlete athletic athletics sport coaches football basketball games teams
hockey team olympic championship soccer coach tennis players boxing player fans champion
contests sporting baseball indoor league
Created July, 2004
22
Sports Figures sports athletes athlete athletic athletics sport coaches football hockey
basketball games teams olympic team coach tennis championship boxing soccer players fans
contests
Games, Hobbies, & Recreation games sports sport athletes hockey basketball golf game
soccer tennis football teams players athlete playing athletic coaches played team athletics
recreation player baseball contests olympic fans championship
Pets pets pet cats dogs dog cat puppy kittens
Performing Arts arts art artistic artists painters sculpture sculptures architecture
Theater theater actors actor playwright playwrights drama plays theatre comedy shakespeare
costumes performers broadway audiences scenery macbeth tragedy scenes theaters performances
stage staged elizabethan script theatres scene audience costume performer performed movie
chorus
Musicians & Composers composers musicians jazz music singers beethoven musician
orchestra musical melody piano tunes symphony concert songs blues opera chord dance song
rhythmic sing rhythm concerts dancing singing tune singer violin dancer guitar rhythms
performances danced instruments sung dancers ballet sang tones instrument band drums chorus
dances chant folk choir instrumental strings
Cinema, Television, & Broadcasting television viewers broadcast broadcasting tv
commercials radio stations entertainment media movies newspapers network news magazines
advertisers networks radios communications studio appearances channels
Music music jazz singers musicians songs composers tunes musical melody piano orchestra
beethoven musician concert song blues symphony sing opera rhythm singing tune dance
rhythmic singer concerts guitar sang dancing rhythms drums sung danced chord violin choir
band chant dancer performances dancers ballet instruments dances folk chorus flute tones strings
instrument harp lessons sings
Dance dance dancing ballet dancer dancers dances music danced jazz musicians piano
composers concert orchestra singers musician opera tunes concerts performances beethoven
songs melody blues musical performers sing symphony costume
Musical Instruments instruments musical instrument music jazz melody piano musicians
musician guitar strings violin orchestra songs concert rhythmic tune rhythm blues song rhythms
sing tones drums opera band singing trumpet singer harp sung
National & Regional Literature national literature regional
Literature & Writing writing writers prewriting essays revising essay writer write draft
narrative readers expository writes assignments autobiography literature descriptive humorous
literary diaries diary compositions novelist
Architecture architecture sculpture art artists artistic statues sculptures sculptor paintings
painters architects renaissance style architect gothic designs painter painting revival artist
portraits styles decoration arts decorative
Created July, 2004
23
Artists artists art painters artist paintings painting painter artistic sculpture portraits sculptures
painted paint portrait architecture museum museums realism renaissance decorative arts sculptor
patrons paints gallery statues florence claude geometric style
Language language spoken dialect languages dialects speak grammatical vocabulary grammar
expressive speaking slang english
Writers & Poets poets poetry poet poems verse prose poem poetic whitman lyric poe writers
rhyme literature epic literary critic novelist lowell bryant imagery dickinson themes imaginative
figurative ballads romanticism novels romantic ironic metaphor hopkins rhythms joyce essays
edgar simile writer
Decorative Arts arts art artistic artists decorative painters artist painting sculpture paintings
painter sculptures architecture designs style
Legends & Folklore legends tales storytellers stories storyteller tale heroes myths story
National & Regional Art art artists artist paintings artistic painting painters painter sculpture
sculptures museum national museums portraits architecture painted
Painting, Drawing, & Graphic Arts sketches drawing drawings artist sketch sketching
pictorial drafting shading draw painting drafters artists art drafter lettering painters painter
graphic artistic geometric paintings portrait portraits drawn dimensions illustrations designer
representations blueprint
Sculpture sculpture sculptures art artists paintings artistic artist painting painters painter
architecture statues painted museum portraits museums statue portrait paint renaissance arts
Periods & Styles periods period styles
Photography photography photographers photographer camera film cameras shutter
photographs aperture photographic photographed films exposure images pictures photograph
studio
Plants plants plant flowering mosses
People in Life Science science sciences biology scientific life geology physics
Medicine medicine medicines doctor doctors prescription sick medical clinic
Invertebrate Animals animals animal zoo zoos lions jellyfish backbones elephants
veterinarian
Created July, 2004
24
Fish fish trout tuna salmon fins fishing aquarium bait swims hook shark bass sharks fishermen
shellfish catch fisherman swim arf perch gills dolphin nets seafood fishes swam carp
Algae & Fungi algae chlorophyll alga fungi protist protists kelp spore multicellular fungus
yeasts protozoans hyphae molds seaweed mushrooms spores chloroplasts microscopic mold
protozoa mushroom phyla pigments rotting ponds parasites green decaying paramecium
decompose
Agriculture, Foodstuffs, & Livestock agriculture agricultural livestock farmers crop crops
wheat subsistence farming cultivation harvest harvesting soybeans farm hogs yields corn farmer
farms grain plows cultivate peanuts surpluses cultivated oats harvested barley export fertilizers
Mammals mammals mammal whales reptiles rodents reptile porpoises
Reptiles & Amphibians reptiles amphibians crocodiles lizards salamanders gills toads turtles
tadpoles frogs lizard snakes vertebrates scales tails fins toad hatch frog fishes eggs
Biological Principles & Concepts concepts principles concept theoretical
Anatomy & Physiology anatomy physiology
Environment environment environments environmental adapted adapt ecology adaptation
surroundings ecologists
Birds birds bird feathers beaks beak wings eagle nest nests loon fly nesting hummingbirds wing
robins geese hawk flew pigeons feather gull crows eagles cranes ostrich swan hummingbird
pigeon owls fluttered flying
Viruses, Monerans, & Protists viruses bacterium bacteria virus protozoa bacterial
protozoans mumps polio viral microbes host microorganisms pathogens protists colds infectious
protist infections infected rabies pneumonia measles influenza microscopic vaccine immune
pasteur immunity infection antibiotics disease villain tuberculosis
History of Asia & Australasia asia asian malaysia strait afghanistan southeast
People in European History history european
People in United States History united states
United States History united states
African History african africans africa nigeria ghana congo tanzania niger ethiopia kenya
guinea history sudan mali sahara tribal
Created July, 2004
25
World History & Concepts world history
Ancient History history ancient historians civilization monuments antiquity civilized
History of the Americas history historians
European History history european europeans
World Cities, Towns, & Villages cities towns suburbs villages metropolitan urban
Regions of the World world regions
Rivers, Lakes, & Waterways rivers lakes streams waterways dumped sewage polluted
dump phosphates dumping reservoirs tributaries erie
Parks & Monuments parks park
Countries countries underdeveloped industrialized nations
Canadian Provinces & Cities cities suburbs metropolitan provinces canadian urban
canadians quebec slums canada
Islands islands island hawaiian coral hawaii mainland philippines honolulu rico puerto
caribbean indies zealand ricans indonesia
Mountain Ranges, Peaks, & Landforms mountain peaks mountains ranges rugged
plateaus slopes sierra valleys mt peak everest foothills rocky ridges plateau climbers highest
appalachians rockies cascade mount steep summit towering himalayas alps landforms nevada
hills andes range
U.S. Cities, Towns, & Villages cities towns suburbs villages metropolitan rural urban
Maps & Mapmaking maps map mapmakers topographic globes mapping symbols contour
atlas symbol mapped elevation
Oceans & Seas oceans ocean seas salinity marine deepest
Exploration & Explorers explorers explorer explored exploration voyages exploring
columbus voyage 1492 sailed explorations magellan cabot expeditions christopher explore
portugal cartier leif claimed expedition isabella riches ferdinand newfoundland adventurers da
U.S. States, Territories, & Regions states united regions
Theology & Practices practices theology religious
Created July, 2004
26
Mythology mythology
Religious Figures religious religion worship christ salvation churches religions church
puritans theology protestants saints puritan bible jesus catholic prayer beliefs protestant catholics
faith teachings ritual preaching reformation christian sin testament christianity christians
preached spiritual congregation preach persecution god hutchinson judaism sabbath orthodox
holy prayers rituals pray sins
Philosophy philosophy philosophical metaphysics plato philosophers philosopher aristotle
socrates thinkers principles disciplines contemplation theology
Religions & Religious Groups groups religious religion religions group protestants christ
christians jesus protestant beliefs catholics teachings christianity christian worship
Scripture sin theology
The Occult
Construction & Engineering construction architectural engineering engineer exterior
architect constructed specifications architects masonry engineers plumbing design building
foundation construct tile projects designers builders
Chemistry chemistry chemist chemists biology physics science sciences chemical
Earth Science earth science geology scientific sciences
Computer Science & Electronics computer computers microcomputer software
computerized computing video graphics electronic diskette capability science calculations
electronics
Machines & Tools machines machine tools wedge tool levers fulcrum lever portable
People in Physical Science physical science sciences physics scientific
Astronomy & Space Science space science astronomy shuttle sciences physicists physics
biology scientific astronaut astronauts
Paleontology
Industry, Mining, & Fuels coal industry mining mines industries mined fuels mills
industrial petroleum
Created July, 2004
27
Physics physics physicists science sciences physicist biology geology chemistry einstein
scientific mathematics
Transportation transportation railroads freight rail highways railroad
Communications communications networks communication telegraph broadcasting
telephone communicating sender stations messages message network receiver radio television
broadcast media
Mathematics mathematics mathematical physics algebra science math
Military Technology military technology technologies
Time, Weights, & Measures measures weights measure measuring measurement metric
measurements measured weight kilogram grams si weighing meter
Military military armed defense strategic generals civilian
Economics & Business business businesses proprietorship entrepreneurs profit
Organizations organizations organization organizational membership organized association
associations
Institutions institutions institutional institution ideology
Political Science political science politics sciences scientific elites
Psychology psychology psychologists psychologist sociology behavior anthropology
scientifically psychological
Law law laws enforced legal enforcement enforce tort statute wrongs
Education education educational schools schooling vocational educators educating colleges
curriculum certification educate universities guidance elementary enrolled training
Anthropology anthropology anthropologist sociology anthropologists disciplines humankind
sociologists sciences mead societies human beings culture scientifically psychology cultures
Military military armed defense strategic generals civilian
Sociology & Social Reform social sociology sociologists sociological sociologist needy
Calendar, Holidays, & Festivals calendar holidays holiday celebrate festival celebrated
celebration
Created July, 2004
28
Archaeology prehistoric anthropologists archaeologists anthropology anthropologist
Sports sports athletes athlete athletic athletics sport coaches football basketball games teams
hockey team olympic championship soccer coach tennis players boxing player fans champion
contests sporting baseball indoor league
Sports Figures sports athletes athlete athletic athletics sport coaches football hockey
basketball games teams olympic team coach tennis championship boxing soccer players fans
contests
Games, Hobbies, & Recreation games sports sport athletes hockey basketball golf game
soccer tennis football teams players athlete playing athletic coaches played team athletics
recreation player baseball contests olympic fans championship
Pets pets pet cats dogs dog cat puppy kittens
Theater theater actors actor playwright playwrights drama plays theatre comedy shakespeare
costumes performers broadway audiences scenery macbeth tragedy scenes theaters performances
stage staged elizabethan script theatres scene audience costume performer performed movie
chorus
Musicians & Composers composers musicians jazz music singers beethoven musician
orchestra musical melody piano tunes symphony concert songs blues opera chord dance song
rhythmic sing rhythm concerts dancing singing tune singer violin dancer guitar rhythms
performances danced instruments sung dancers ballet sang tones instrument band drums chorus
dances chant folk choir instrumental strings
Cinema, Television, & Broadcasting television viewers broadcast broadcasting tv
commercials radio stations entertainment media movies newspapers network news magazines
advertisers networks radios communications studio appearances channels
Music music jazz singers musicians songs composers tunes musical melody piano orchestra
beethoven musician concert song blues symphony sing opera rhythm singing tune dance
rhythmic singer concerts guitar sang dancing rhythms drums sung danced chord violin choir
band chant dancer performances dancers ballet instruments dances folk chorus flute tones strings
instrument harp lessons sings
Dance dance dancing ballet dancer dancers dances music danced jazz musicians piano
composers concert orchestra singers musician opera tunes concerts performances beethoven
songs melody blues musical performers sing symphony costume
Musical Instruments instruments musical instrument music jazz melody piano musicians
musician guitar strings violin orchestra songs concert rhythmic tune rhythm blues song rhythms
sing tones drums opera band singing trumpet singer harp sung
Created July, 2004
29
9 Appendix B: Sample user goals for Online Encyclopedia
131 words
0.83 cosine with full encyclopedia article
Find encyclopedia article about Audiometer
Audiometer, instrument for testing hearing. The audiometer is an essentially simple instrument
that produces pure tones of various fixed pitches (frequencies) heard through headphones.
Hearing is tested one ear at a time. The operator can switch between frequencies and repeat the
process with each frequency. Typically, sensitivity may be tested at frequencies of 125 hertz (Hz,
or cycles per second), 250 Hz, 500 Hz, 1000 Hz, 2000 Hz, 4000 Hz, 8000 Hz, and 12,000 Hz. As
an alternative to testing the normal mode of hearing through headphones, hearing by bone
conduction can be tested. Hearing is never uniform over all frequencies and commonly varies
widely at different frequencies. Internally, audiometers consist of a transistorized, variablefrequency audio oscillator—usually a simple feedback device—capable of producing a
sinusoidal (near sine-wave) output.
142 words
0.84 cosine with full encyclopedia article
Find encyclopedia article about Belemnite
Belemnite, extinct group of marine animals that resembled squid in form and probably behavior,
but possessed a unique internal shell, which was easily fossilized. The largest fossilized
belemnite shells are about 2 m (about 6 ft) long and about 5 cm (2 in) wide. Belemnites were
similar in shape to squid. The body consisted of a torpedo-shaped head and ten tentacles. The
head contained the shell and all internal organs. As with squid, belemnites possessed a beaklike
mouth for eating, and an ink sac. Belemnites evolved from a primitive ancestor with an external
shell. As with squid, rapidly swimming belemnites sometimes suffered damaging collisions, as
evidenced by numerous fossils of broken shells that appear to have healed. Identification of
individual belemnite species in a rock layer enables paleontologists to date the rock layer.
Belemnites are in the phylum Mollusca and class Cephalopoda.
109 words
0.83 cosine with full encyclopedia article
Find encyclopedia article about Cybernetics
Cybernetics, interdisciplinary science dealing with communication and control systems in living
organisms, machines, and organizations. Cybernetics developed as the investigation of the
techniques by which information is transformed into desired performance. One of the basic
tenets of cybernetics is that information is statistical in nature and is measured in accordance
with the laws of probability. The measure of probability is known as entropy (see Information
Theory; Thermodynamics). Purposive behavior in humans or in machines requires control
mechanisms that maintain order by counteracting the natural tendency toward disorganization.
Cybernetics has also been applied to the study of psychology, servomechanisms, economics,
neurophysiology, systems engineering, and the study of social systems.
185 words
0.88 cosine with full encyclopedia article
Find encyclopedia article about Fern
Created July, 2004
30
Fern, common name for any of a division of cryptogamous (spore-producing) plants. Ferns are
found throughout the world. Most grow in damp, shady places, although certain species grow on
dry ground, soil, or rocks. Ferns are among the oldest land plants. Tree ferns have woody trunks
without branches, topped with clusters of feathery leaves, or fronds. Most ferns, however, have
no trunks, and fronds grow directly from a short underground stem. The asexual, or sporophyte,
generation represents the fern plant as it is commonly known. There are two major groups of
ferns, leptosporangiate and eusporangiate. In leptosporangiate ferns, the sporangium develops
from the outer derivative of a single epidermal cell, is slender-stalked, and produces fewer than
64 spores. In eusporangiate ferns, the sporangium develops from the inner derivatives of several
epidermal cells, is sessile, or thin-stalked, and produces more than 256 spores. The prothallium,
which in no way resembles the asexual fern plant, is a small, flat, heart-shaped structure with a
number of rhizoids growing on its underside. Ferns used as pot plants are usually tropical
species. Scientific classification: Ferns make up the division Filicinophyta.
155 words
0.92 cosine with full encyclopedia article
Find encyclopedia article about Ferreting
Ferreting, use of ferrets to flush rats or rabbits from underground channels and paths. It is
believed that the domestic ferret was derived from the European polecat, Mustela putorius, as an
aid in hunting rabbits. From North Africa, the Romans presumably carried the ferret and the
pastime of ferreting to Europe and elsewhere. Female ferrets, because they are smaller than
males and able to pursue their prey through smaller passageways, are preferred for ferreting.
Ferrets are still used in Europe for reducing rodent populations on farms and in England for
hunting rats along hedgerows. In ferreting for rabbits, the ferret is sometimes muzzled to prevent
it from killing a trapped rabbit underground. Ferrets are able to work at seven weeks old. They
hunt instinctively and without training, but it has long been a custom in many areas to cross
breed ferrets with wild polecats on the assumption that such crosses produce more vigorous
ferrets.
103 words
0.90 cosine with full encyclopedia article
Find encyclopedia article about Frequency Modulation
Frequency Modulation (FM), system of radio transmission in which the carrier wave is
modulated so that its frequency varies with the audio signal being transmitted. FM broadcasting
stations can be operated in the very-high-frequency bands at which AM interference is frequently
severe; commercial FM radio stations are assigned frequencies between 88 and 108 Mhz. In
2000, there were about 5,770 FM stations. In 1961 the Federal Communications Commission
authorized FM stereophonic broadcasting. Thereafter, the FM band drew increasing numbers of
listeners to popular as well as classical music, and commercial FM stations began to draw higher
audience ratings than AM stations.
205 words
0.82 cosine with full encyclopedia article
Find encyclopedia article about Hmong
Hmong, minority ethnic group that lives primarily in China and Southeast Asia. About 2 million
Hmong live in Southeast Asian countries, such as Vietnam, Laos, Thailand, and Myanmar.
Created July, 2004
31
Another 10 million Hmong live in the southern provinces of China. The United States has the
largest Hmong refugee community, with a population of about 300,000 in 2001. The word
Hmong, which means “man” in the Hmong language, is the name used by the Hmong people
themselves. During the Vietnam War, some Hmong began translating the name Hmong as “free
man” to express their desire for political independence. The Hmong language contains seven
tones. Within Hmong society, subgroups speak slightly different versions of the Hmong
language. The largest subgroups are White Hmong, Red Hmong, Blue or Green Hmong, and
Striped Hmong. A Hmong bride joins the clan of her husband. With French encouragement,
many Hmong turned to opium cultivation during World War II (1939-1945).
Hmong in the United States. Between 1975 and 1994, more than 110,000 Hmong refugees
resettled in the United States. Because Hmong tend to have large families, these communities
have grown rapidly. Hmong families have faced considerable challenges in adapting to
American life. Hmong women have earned money selling their colorful needlework.
Created July, 2004