Playing Tangram with a Humanoid Robot

Playing Tangram with a Humanoid Robot
Jochen Hirth, Norbert Schmitz, and Karsten Berns
Robotics Research Lab, Dept. of Computer Science, University of Kaiserslautern, Germany
j_hirth,nschmitz,berns@{informatik.uni-kl.de}
Abstract
An open question in the area of social robot interaction is how to design test scenarios that on one hand provide the
required complexity and on the other hand are still describable. The tabletop game Tangram has been selected as a test
scenario for human-robot interaction. This paper describes how the required skills, to enable the humanoid robot ROMAN
to play the game, have been ralized. Experimental results demonstrate the functioning of the system and show how
emotion influences robot’s motivation.
1
INTRODUCTION
For recent years, there has been a lot of research in the
area of social interactive robots. Starting from the work of
C. Breazeal [1] to the latest developments in the area of
interactive robots as they are described in [2] or [3]. Currently most of the roboticists dealing with social interacting robots agree that an emotion component increases the
performance of an interactive robot. Therefore, it is no
question that an interaction situation for testing the interactive capabilities of a robot requires a minimum of complexity. The following requirements for the test scenario
have been figured out:
• Enough space of action, to enable various actions
and behaviors of the human being as well as of the
robot.
• Limited complexity, so that all the required information can be perceived and the scenario can be described.
• A minimal length of the interaction process, so that
changes of the emotional state and of the motivation
of the interaction partners can/will happen.
Since scenarios like ticket window situations do not fulfill
these requirements, the decision was made to realize the
tabletop game Tangram as testing scenario. The game consists of shadow image presented to the player, which he
has to be form using the puzzle pieces of simple geometric
shapes.
The complexity and observability of the Tangram allows
both the creation of complex or even unsolvable problems as well as the analysis of the game situation and the
progress of the interaction partner.
Figure 1: The humanoid robot ROMAN playing the Tangram game with a human interaction partner.
Interactive human computer scenarios with tabletop applications are available for a broad set of games. In [4] the
realization of an interactive memory card game for a humanoid robot is described. Recently Verdie [5] used Tangram as interactive tabletop application and analyzed the
position and orientation of the game pieces with an overhead camera. He extracted edges and corners of each piece
and generated a tracking system to follow modifications of
the game situation. Although this analysis is very accurate and provides a complete dataset for further analysis it
is not transferable to a humanoid robot. Neither the precondition of a non-moving camera and table game nor the
assumption of constant lighting conditions can be met in
interactive human-robot scenarios.
This paper provides detailed information how the humanoid robot ROMAN, see Fig. 1, has been enabled to play
the Tangram game. It is explained how the necessary skills
have been implemented within the emotion-based architecture described in [6]. Furthermore, the results of an experiment to demonstrate the influence of emotion on the
robot’s motivation are discussed. The paper is arranged as
follows: At first, a brief introduction to the emotion-based
architecture that forms the basis of ROMAN’s control system is given. Afterwards, the implementation of the components that are mainly involved in the Tangram game is
described in detail. In Sect. 4 the experimental results are
presented. Finally a conclusion and an outlook for future
work are given.
Emotion
Center
rating
function
selective
function
regulative
function
Motives
expressive
function /
motivational
function
Appraisal
Percepts
sensor
data
Sensors
Behaviors
direct responses
control
parameters
Actuators
Figure 2: The structure of the emotion-based architecture
that controls the interactive behavior of the robot ROMAN.
2
CONTROL ARCHITECTURE
As described in [7] emotion plays an important role for
humans’ social behavior. Therefore, the control system
of ROMAN is realized as an emotion-based architecture,
cf. [6]. The system consists of 4 main components (Fig. 2),
percepts, motives, appraisal system, and behaviors, that
enable ROMAN to act and behave in real world interaction situations. The perception system, percepts, collects
information from the environment. These information e.g.
face detector or color detector are combined using different fusions. That way a dynamic model of the interaction
partner and the current situation can be created. [8] The
appraisal system uses this information as well as the current internal state to evaluate the situation and derive an
emotional state. The internal situation is represented by
the previous emotional state and by the satisfaction of the
different motives. Each motive represents a specific goal
of ROMAN. The behavior of a motive can be explained using the control loop theory. Depending on the current state
and on the target state a motive calculates its satisfaction
and its activity. This activity is used to stimulate different
actions. The current emotional state influences the calculation of the motives satisfaction and the robot’s expressive behavior. The behavior system provides 4 main parts.
The lowest layer is formed by so called “basic behaviors”,
which are directly derived from the robot’s actuation system and represent motion primitives, like turn head left or
turn head right. On the next layer actions and responses
are realized by combining different basic habits. That way
complex behavior like focusing the interaction partner (actions) are implemented. The highest layer is formed by the
scenarios. A scenario provides information in which order
the different complex habits need to be stimulated to reach
a certain goal. The scenarios for their part are stimulated
by the corresponding motives.
3
IMPLEMENTATION
This section provides detailed descriptions how the capabilities to play Tangram have been realized within the
emotion-based architecture. It is pointed out which information need to be perceived and how the perceived information can be combined to get more complex information
on the current situation. It is described which actions are
required to play Tangram and how the Tangram scenario is
implemented. Furthermore, details on the motivation system that causes ROMAN to play Tangram are explained.
General information on the definition of the single components of the emotion-based architecture can be found
in [6], whereas this section provides a detailed description
of a specific example.
3.1
Perception System
The perception system of ROMAN consists of 2 stereo camera systems and 6 microphones. That way ROMAN is able
to detect and focus its interaction partners by image processing and using the microphones ROMAN can estimate
the position of a sound source and figure out whether the
sound source is a human voice or not. Furthermore, RO MAN is able to detect the Tangram board and to evaluate
the current state of the game with the help of its cameras.
All sensors are included in the robot itself and no external
sensors are necessary to handle the Tangram game situation. In the terminology of the UKL Emotion-based Architecture, every component that represents specific information is called percept.
The components mainly used for the Tangram scenario are:
Face Detector, Skin detector, Stereo Processing, and Tangram Detector. The Face Detector is responsible for the
detection of face candidates based on a previously trained
face model. The combination of Face Detector, Skin detector, and Stereo Processing is used to determine the current
position of a human in the robot’s environment. In this
particular situtation the position information is used to decide whether a human is categorized as “Tangram Player”
or not.
To analyze the Tangram game from the robot’s point of
view requires visual analysis from a moving camera and
unknown position and orientation towards the interaction
area. Therefore, the robot must be able to detect the board
and correct any distortions. The implemented approach
simply uses color information to find and evaluate the Tangram board. Geometric primitives are used to correct perspective warping.
3.2
Tangram Motive
As already mentioned, within the UKL Emotion-based Architecture each motive represents a specific goal. The motives calculate their satisfaction depending on the achievement of these goals. Depending of the satisfaction value
the motives state can be classified as “undersatisfied”, “satisfied”, “oversatisfied”. Each motive’s goal is always to
reach a satisfied stated.
In case of the Tangram motive undersatisfaction means that
the robot wants to play the game. The satisfaction value of
the motive is increases if a human being plays the Tangram
game with robot otherwise it is decreased. Therefore, it
will stimulate behaviors that motivate persons to play the
game or to go on playing. The longer the game lasts the
more increases the satisfaction. After a while the motive
becomes oversatisfied – boredom is experienced. In this
state the motive will stimulate behaviors to make the human Tangram player quit the game.
troller tries to keep the satisfaction in a medium state. If
the satisfaction is below a lower threshold or over an upper threshold specific behaviors are stimulated to reach a
medium level of satisfaction. The principle functioning of
a motive is depicted in Figure 3.
3.3
The Behavior System
Get new
player
if game started
timer up
Time
Focus player
Focus board
Satisfaction Level
Emotional State
timer up /
s.th. changed
game rated
Motive
-1
Satisfying Event
0
1
Rate game
Lower
Upper
Threshold Threshold
Stimulation of
Corresponding Behaviors
Environment
Figure 3: The functioning of a motive as a control loop:
Depending on the presence of the satisfying event and on
the emotional state the satisfaction of the motive is calculated. If this is below or above specific thresholds, the
corresponding behaviors are stimulated in order to reach a
satisfied state again.
The emotional state influences the motivation in a way that
if positive emotions are experienced while fulfilling a task
this task is performed for a much longer time. On the
ohter hand if negative emotions are experienced the task
is stopped earlier. Furthermore, the desire to perform tasks
related to positive emotions again is much higher than for
tasks related to negative emotions. For the realization of
this phenonemon two limits, an upper limit and a lower
limit have been intorduced. The range below the lower
limit represents undersatisfaction between the limits satisfaction and above the upper limit oversatisfaction. These
limits are changed depending on the emotional state. A
rather positive emotional state will increase both limits; a
negative state will decrease the limits. That way it takes
much longer to reach an oversatisfied state if positive emotions are experienced. If negative emotions are experienced the upper limit will be reached very early. Furthermore, if positive emotions have been experienced the lower
limit is much higher. That means this motive will much
faster reach an undersatified state and therefore become
active again. On the other hand if negative emotions are
experienced the lower limit will be quite low and it takes
much longer time until the motive becomes active again.
The functioning of a motive can be described as a control
loop. The control variable is the satisfaction and the con-
Figure 4: The Tangram scenario: The chain of actions that
need to be executed in order to play Tangram is represented
as a finite state machine.
The actions, complex habits, which are used for playing
Tangram, are: Look at Point, that enables the robot to focus a 3D-point in its environment and Search Object, that
allows ROMAN to look for a specific object in its environment. Furthermore, the so called Speech Centre is used,
which can be regarded as a special ability to act – a complex habit. Besides the realization of verbal communication, the Speech Centre also activates speech adapted nonverbal expressions. As described in Sect. 2, the highest
layer of the habit system, the so called Scenarios, provide
information about the activation order of the different actions. This is realized by a finite state machine, where the
single nodes provide information which action to stimulate, see Fig. 4. In order to play Tangram, ROMAN tries
to find a human Tangram player. ROMAN is looking for a
human being in its environment and asks the human being
to play Tangram. If the human being starts playing Tangram, ROMAN will focus the player and after a while or
if changes have been detected, it will look at the Tangram
board, generate a rating of the current status of the game
and on the performance of the human player. Depending
on this rating ROMAN will give some comments to the human player. The runtime of the timer can vary. They depend on the current emotional state, more precisely on the
arousal value, of ROMAN. The more aroused the shorter
the timer runtime.
By implementing the scenarios in this way, new scenarios
can be realized without lots of effort, since the structure
is specified in the scenarios and the content in the actions.
For the realization of a new scenario only the structure, a
new finite state machine that specifies the stimulation order
of the actions, must be implemented.
4
4.1
EXPERIMENTS
Playing Tangram
The setup for the experiments was as follows: The Tangram board was placed on the top of a table located in front
of the ROMAN. A human being was told to enter the scene
and play Tangram, see Fig. 1. During the experiments the
robot was operating autonomously without any manual intervention. To evaluate the system a test run of about 5 min
has been recorded. Throughout the test run, the activities
of the percepts and the habits have been recorded. In the
following an extract of about 2.5 min is discussed in detail.
For the final paper the results of questionaire evaluation of
the robot’s behavior while playing the Tangram game will
be presented.
ment shows that the perception system is able to extract the
0
two basic percepts Ptangram and Pperson
. Furthermore, it
shows that if the human tracking is lost reinitialization occurs. It also shows that the realized habits enable the robot
to handle the interaction situation.
(a) 65s : At the beginning with one correct piece
(b) 82s : Tracking is lost
(c) 95s : Reinitialized tracker
Figure 5: This plot shows the activity of the actions involved in the Tangram game, while ROMAN is playing the
game.
The behavior of the robot is stimulated by the "Tangram
Motive" as described in Section 3.2. When the motive is
unsatisfied it stimulates the Tangram scenario and robot
tries to find a human being willing to play the game. The
activities of the actions of the Tangram scenario during a
Tangram game, are depicted in 5.
Here two gaps at 80 and 110 seconds are visible. During
these periods the tracking of the human interaction partner is lost due to the fact that the face is turned towards
the Tangram board. After approx. 45 s ROMAN detects a
human being.
Therefore, Get new player gets active and ROMAN asks
the human being to play Tangram. At the same time Focus
player gets active and ROMAN focuses the possible Tangram player. The human being starts to play the game and
after approx. 62 s Rate game gets active, stimulates Focus
board and ROMAN starts to evaluate the current solution.
Afterwards, ROMAN tells the human player which tiles are
placed correctly, and so on. During the whole process of
playing Tangram it can be seen, that Focus player is inhibited when Focus board gets active. Furthermore, in some
situations (after 102 s and 160 s) Rate game is longer active than Focus board. This is because the whole process
of rating the game consists of looking at the board, evaluating the current state, and telling the human being about
the current state.
Figure 6 depicts a top view, body view and eye view image
captured during playing the Tangram game. The experi-
(d) 162s : Puzzle solved
Figure 6: Image sequences captured during the test run.
The tracking of the interaction partner is shown as well as
the rating of the Tangram situation. The top view image is
given as reference.
4.2
Influencing the Robot’s Motivation
A typical behavior that can be observed by humans is
that positive emotions increase the motivation to continue
the currently performed task, while negative emotions decrease the motivation and the humans give up the tasks
much faster. As described in before the emotional state
influences the calculation of the motives satisfaction by
shifting the tresholds. If the robot is in a positive emotional state it will take much longer until the motive becomes over stimulated – bored – and if negative emotions
are experienced the thresholds will be decreased and the
over stimulation will be reached much faster.
To test this function the Tangram game is played twice. In
the first run positive emotions are provoked in the second
run negative ones.
To have the robot experiencing positive emotions the interaction partner places the tiles correclty and the time between placing two tiles has been chosen rather small. If the
Tangram motive of the robot is active, the robot wants to
play the game; it should now experience positive emotions.
On the other hand, to bring the robot in a negative emo-
tional state the Tangram player places the tiles wrongly and
also takes a lot of time until placing the next tile. Therefore, the robot’s emotions should become more negative.
This should lead to a faster over stimulation and the robot
will try to end the game. The time until the robot wants to
quit the game is meassured.
Arousal
Valence
Stance
Emotional
State
1
0.5
0
−0.5
−1
0
100
200
300
400
500
600
300
400
500
600
Tangram Motive
Target Rating
1
0.5
0
0
100
200
Time [s]
Figure 7: The activity of the Tangram Motive while the
robot experiences positive emotions. The target rating indicates the motivation of the robot to play the game (0 =
high motivation, 1 = low motivation).
Arousal
Valence
Stance
Emotional
State
1
0.5
It can be seen that the emotional state starts from 0. After a few seconds the target rating falls from 1 to 0 since
the motives desire is satisfied. Directly after the decrease
of the target rating, which has been caused by the human
playing the game, both valence and stance start increasing and raising to 1. The incease of valence is caused by
the satisfaction of the motive, the increase of stance by the
perception of the desired stimulus – somone is playing the
game with the robot. The big difference when comparing
the two diagrams of the emotional state lies in the shape of
the curve representing the arousal value. While in Figure 7
the arousal value is increased in single steps and it sloly
decreases between these step, in Figure 8 the arousal value
constantly decreases. This different behavior of the arousal
value is caused by the Tangram player. In the first case the
Tangram player is very active and places the Tangram tiles
correctly. Everytime the robot recognizes an action of the
player in order to improve the current state of the game the
arousal is increases. In the second case the player does not
try to solve the game.
0
−0.5
−1
0
50
100
150
200
100
150
200
Target Rating
Tangram Motive
absolute value of the satisfaction. It can be regarded as a
meassurement for boredom. If the target rating reaches 1
the robot wants to stop the currently performed task. The
results of the experiment are depicted in Figure 7 and Figure 8. In both situations the robot started in a neutral emotional state, arousal, valence, and stance were all equal to
0. Since the Tangram player is already present in the scene
her presence does not affect the aroual value. That means
the only way she can increase the arousal is by placing the
puzzle pieces.
1
0.5
0
0
50
Time [s]
Figure 8: The activity of the Tangram Motive while the
robot experiences negative emotions. The target rating indicates the motivation of the robot to play the game (0 =
high motivation, 1 = low motivation).
The robot’s emotional state is represented using the 3 categories arousal, valence, and stance, where arousal describes how thrilling a stimulus is, valence describes the
satisfaction of the robot in the current situation, and stance
describes how desired a perceived stimulus is in the current situation. Furthermore, the target rating of the Tangram motive is plotted. Internally it is calculated as the
Therefore, the intensity of the Tangram game is not increased and since no other person enters the scene and
no loud sounds have been perceived the arousal value decreases constantly. As already mentioned the absence of
any arousing stimuli leads to boredom. This phenomenon
is considered in the calculation of the motives satisfaction.
Low arousal decreases the thresholds for undersatisfaction
and overstatisfaction. The lower the threshold for oversatisfaction the earlier the corresponding oversatisfied state,
which means that the robot is tired of its currently perfomed task. On the other hand if the current situation entertains the robot and provides interesting stimuli – represented by high intensity – the threshold for overstimulation
will increase and therefore it will take much longer time
until the robot becomes bored. This behavior can be seen
when comparing the two diagrams for the Tangram Motive. The decrease of valence is caused by the unsatisfied
state of the motive, the decrease of stance is caused by the
fact that there is still someone playing the game although
the robot wants to quit. In the first case (Figure 7) the emotional state changes the threshold for oversatisfaction in a
way that the motivation to play the games lasts for 612.7 s,
while in the second case (Figure 8) the robot looses its motivation already after 135.04 s.
Pos. Emotion
Neg. Emotion
Time [s]
800
600
400
200
0
0
20
40
60
80
100
Test Run #
Figure 9: For every test run the time until the robot looses
its motivation to play the Tangram game is plotted, once
for the case where the Tangram player tries to provoke positive emotions (red dots) for the robot and once for the case
where negative emotions are induced (blue dots).
To ensure the consistancy of the described robot behavior, this experiment has been repeated 100 times in simulation. The results of these test runs are depicted in Figure 9. For every test run two dots are plotted. The red
one indicates the time until the robot looses its motivation
to play the game in the cases that the player tries to motivate the robot, the blue dot represents the case where the
player tries to demotivate the robot. It can be seen that
there has been a significant different between the motivated
case and the demotivated case. It is obvious that when the
robot experiences positive emotion the variation is much
higher than in the case where negative emotions are experienced. This is because motivation of the robot depends on
the progress in the game. Therefore, it depends on the puzzle pieces placed by the Tangram player and on the recognition of these pieces by the robot. Since these parameters
are not constant throughout the interaction the achieved results varie. In the case where the robot should be demotivated the player does not change the puzzle pieces. Therefore, this parameter does not affect the robot’s emotional
state and therefore there is less variation in these results.
The achieved results show that the realized architecture
implements a major aspect of the motivational characteristic of emotion. Positive emotions increased the robot’s
motivation to continue the currently performed task significantly compared to negative emotions that lead to the
opposite behavior. The mean time of all test runs while
experiencing positive emotions until the robot looses its
motivation comes up to 683.6111 s, while the mean time
for the negative emotional case is 140,0602 s.
5
CONCLUSIONS AND FUTURE
WORKS
This paper provided detailed information how to implement the necessary skills for enabling the humanoide robot
ROMAN to play Tangram. The whole implementation is
realized within the UKL Emotion-based Architecture described in [6]. This architecture is especially designed for
controlling interactive robots. To evaluate the functioning
of the implemented components experiment results have
been introduced that prove that ROMAN is able to handle
the Tangram scenario. Using this scenario an experiment
has been conducted that shows the influence of emotion
to the robot’s motivation. For the future further experiments using the Tangram scenario will be conducted to
investigate the influence of emotion to human-robot interaction. For example whether the human interaction partner is able to deduce the robot’s motivation depending on
the robot’s interactive and expressive behavior or contrary
to the experiment presented in this paper it can be investigated whether the robot can influence the motivation of
its interaction partner by changing its own emotional reactions. Appart from this, further interaction scenarios
should be realized to test whether the results achieved for
the Tangram scenario can be transfered.
References
[1] C. Breazeal, “Sociable machines: Expressive social
exchange between humans and robots,” Ph.D. dissertation, Massachusetts Institute Of Technology, May
2000.
[2] J. Saldien, K. Goris, B. Vanderborght, J. Vanderfaeillie, and D. Lefeber, “Expressing emotions with the
social robot probo,” Int. J. of Social Robotics, vol. 2,
no. 4, pp. 377–389, 2010.
[3] H. Kozima, M. P. Michalowski, and C. Nakagawa,
“Keepon – a playful robot for research, therapy, and
entertainment,” Int. J. of Social Robotics, vol. 1,
no. 1, pp. 3–18, 2009.
[4] V. Ng-Thow-Hing, J. Lim, J. Wormer, R. K. Sarvadevabhatla, C. Rocha, K. Fujimura, and Y. Sakagami, “The memory game: Creating a human-robot
interactive scenario for asimo,” in 2008 IEEE/RSJ
Int. Conf. on Intelligent Robots and Systems, Nice,
France, September, 22-26 2008, pp. 779–786.
[5] Y. Verdie, “Surface gesture & object tracking on
tabletop devices,” Master’s thesis, Virginia Polytechnic Institute and State University, 2010.
[6] J. Hirth, N. Schmitz, and K. Berns, “Towards social robots: Designing an emotion-based architecture,” Int. J. of Social Robotics, 2011, dOI:
10.1007/s12369-010-0087-2.
[7] H. Hobmair, S. Altenhan, S. Betscher-Ott, W. Dirrigl,
W. Gotthardt, and W. Ott, Psychologie. Troisdorf,
Germany: Bildungsverlag EINS, 2003.
[8] N. Schmitz, “Dynamic Modeling of Communication
Partners for Socially Interactive Humanoid Robots”,
Dissertation, University of Kaiserslautern, 2011.