catch fish: virtual balancing game to enhance trunk

CATCH FISH: VIRTUAL BALANCING GAME TO ENHANCE TRUNK
STRENGTH FOR STROKE PATIENT
KHALIL ASYRANI BIN SULAIMAN
UNIVERSITI TEKNOLOGI MALAYSIA
PSZ 19:16 (Pind. 1/07)
UNIVERSITI TEKNOLOGI MALAYSIA
DECLARATION OF THESIS / UNDERGRADUATE PROJECT REPORT AND COPYRIGHT
Author’s full name : KHALIL ASYRANI BIN SULAIMAN
Date of Birth
: 24TH MARCH 1991
Title
: CATCH FISH: VIRTUAL BALANCING GAME TO ENHANCE TRUNK STRENGTH
FOR STROKE PATIENT
Academic Session : 2013/2014
I declare that this thesis is classified as:
CONFIDENTIAL (Contains confidential information under the Official Secret Act
1972)*
RESTRICTED
(Contains restricted information as specified by the
organization where research was done)*
OPEN ACCESS
I agree that my thesis to be published as online open access
(full text)
I acknowledged that Universiti Teknologi Malaysia reserves the right as follows:
1. The thesis is the property of Universiti Teknologi Malaysia
2. The Library of Universiti Teknologi Malaysia has the right to make copies for the
purpose of research only.
3. The Library has the right to make copies of the thesis for academic exchange.
Certified by:
SIGNATURE
SIGNATURE OF SUPERVISOR
KHALIL ASYRANI BIN SULAIMAN
DR. EILEEN SU LEE MING
(910324-10-5937)
Date: JUNE 2014
NOTES:
*
Date: JUNE 2014
If the thesis is CONFIDENTAL or RESTRICTED, please attach with the letter from
the organization with period and reasons for confidentiality or restriction.
“I hereby declare that I have read this thesis and in my/our* opinion this thesis is
sufficient in terms of scope and quality for the award of the degree of Bachelor of
Engineering (Computer)”
Signature
: ………………………….........
Name of Supervisor
: Dr. Eileen Su Lee Ming
Date
: ………………………………..
CATCH FISH: VIRTUAL BALANCING GAME TO ENHANCE TRUNK
STRENGTH FOR STROKE PATIENT
KHALIL ASYRANI BIN SULAIMAN
A thesis submitted in fulfillment of the
requirements for the award of the degree of
Bachelor of Engineering (Computer)
Faculty of Electrical Engineering
Universiti Teknologi Malaysia
JUNE 2014
ii
DECLARATION
I declare that this thesis entitled “Catch Fish: Virtual Balancing Game to
Enhance Trunk Strength for Stroke Patient” is the result of my own research
except as cited in the references. The thesis has not been accepted for any
degree and is not concurrently submitted in candidature of any other degree
Signature
: ………………………………………....
Name of Candidate
: KHALIL ASYRANI BIN SULAIMAN
Date
: …………………………………………
iii
Specially dedicated to Mama and Abah
I really miss both of you.
May Allah bless you both always.
iv
ACKNOWLEDGMENT
Firstly, I would like to give the greatest gratitude to Dr.Eileen Su Lee Ming
from Electrical Faculty of University Teknologi Malaysia for the guidance in
completing my project – “Catch Fish: Virtual Balancing Game To Enhance Trunk
Strength For Stroke Patient”. It would be impossible to complete without helps,
advices and hospitality given by her.
In addition, I would like to thank Dr. Yeong Che Fai and Yong Bang Xiang for
helping me in providing libraries and source codes to aid me in designing the algorithm
of this project.
Apart from that, I also would like to give my appreciation to all my friends,
specifically Ahmad Bustamam, Chang Mun Keong, Muhammad Safwan and
Muhammad Fadzly for helping me in testing, giving idea, and feedbacks for this
project.
Lastly, for the others that had helped me in this project, your help is appreciated.
v
ABSTRACT
Stroke is a common yet serious medical condition that potentially can affect
every person. Due to the danger of stroke, lots of prevention and awareness
campaign have been broadcasted throughout the media around the world. However,
stroke is not a disability or disease that cannot be cured. Lots of rehabilitation centre
has been opened to aid the stroke patient. With rehabilitation, stroke survivors can
regain some form of functionalities. Many alternatives are available to help stroke
patients in rehabilitation, either through conventional method or modern alternatives
to further enhance and improve the conventional methods used today. This project
entitled, “Catch Fish: Virtual Balancing Game to Enhance Trunk Strength for Stroke
Patient” aimed to develop an interactive game to help stroke patients in their training.
This project used the concept of virtual reality as the main idea to motivate stroke
patients in their training. Microsoft Kinect Xbox 360 Sensors and XNA Game
Development kit are used to develop the game. This project can help to enhance the
stroke patient trunk strength through single player or multiplayer gameplay.
vi
ABSTRAK
Strok merupakan penyakit kronik yang umum dan serius, berpotensi
menyerang setiap orang. Dengan kesedaran akan bahaya strok, banyak kempen
kesedaran dan pencegahan awal telah disiarkan di media di seluruh dunia. Walau
bagaimanapun, strok bukan kecacatan atau penyakit yang tidak boleh diubati. Banyak
pusat pemulihan telah dibuka untuk membantu pesakit strok. Melalui proses
pemulihan, mangsa strok boleh mendapatkan semula fungsi-fungsi motor utama pada
badan mereka. Banyak alternatif yang sedia untuk membantu pesakit strok dalam
proses pemulihan, sama ada melalui kaedah konvensional atau alternatif moden yang
membantu meningkatkan dan membaiki kaedah konvensional yang digunakan masa
kini. Projek ini yang bertajuk, "Catch Fish: Virtual Balancing Game to Enhance Trunk
Strength for Stroke Patient " bertujuan untuk pembangunan permainan interaktif untuk
membantu pesakit strok dalam latihan mereka. Projek ini menggunakan konsep
“virtual reality” sebagai idea utama untuk memberi motivasi kepada pesakit strok
ketika sesi latihan mereka. Microsoft Kinect Xbox 360 Sensor dan XNA Game
Development kit digunakan untuk pembangunan permainan ini. Projek ini boleh
membantu untuk meningkatkan kekuatan tulang belakang pesakit strok melalui
permainan perseorangan atau permainan berkumpulan.
vii
TABLE OF CONTENTS
CHAPTER
1
2
TITLE
PAGE
DECLARATION
ii
DEDICATION
iii
ACKNOWLEDGMENT
iv
ABSTRACT
v
ABSTRAK
vi
TABLE OF CONTENTS
vii
LIST OF FIGURES
x
LIST OF TABLE
xii
LIST OF ABBREVIATIONS
xiii
LIST OF APPENDIX
xiv
INTRODUCTION
1
1.1
Project Background
1
1.2
Problem Statements
2
1.3
Research Objective
3
1.4
Research Scopes
4
LITERATURE REVIEW
5
2.1
Conventional Rehabilitation Therapy
5
2.2
FAME (Fitness and Mobility Exercise)
6
viii
2.3
11
2.5
Balloon Burst
13
2.8
5
6
8
Serious Games for Rehabilitation
2.7
4
Balance Exercise
2.4
2.6
3
Neverball: Virtual Environment Game-Based
PlayMancer: A Serious Gaming 3D
Environment
Rehabilitation Gaming System (RGS)
Catch the Fish: Virtual Balancing Game to
Enhance Trunk Strength for Stroke Patients
14
16
18
RESEARCH METHODOLOGY
19
3.1
Hardware
19
3.2
Software
23
3.3
Boat Game
32
RESULTS AND DISCUSSION
29
4.1
Game Logic and In-Game Assessment
29
4.2
Analysis and Data Extraction
33
CONCLUSION AND FUTURE
RECOMMENDATIONS
34
5.1
Introduction
34
5.2
Catch Fish Game
34
5.3
Limitations
35
5.4
Recommendations
36
PROJECT MANAGEMENT
37
6.1
Introduction
37
6.2
Project Schedule
37
REFERENCES
40
ix
APPENDIX A
45
List of Audio
45
x
LIST OF FIGURES
FIGURE NO.
TITLE
PAGE
2.3.1
Neverball in-game interface and gameplay
8
2.3.2
Neverball played on a wobble board
9
2.5.1
System setup for Gamebased Exercises for Dynamic Short-
13
Sitting Balance Rehabilitation [28]
2.6.1
Simplified block diagram of PlayMancer platform [30]
15
2.7.1
The Rehabilitation Gaming System set-up [32]
16
2.7.2
Virtual player’s arm tried to “hit” the flying sphere [32]
17
2.8.1
Catch Fish game menu
18
3.1
Kinect sensors (1) Infra-red (IR) Depth Sensors, (2) IR
19
Camera, (3) Microphone array, (4) Tilt sensor and motor
3.2
Microsoft Kinect camera and microphone used to measure
20
object’s depth
3.3
Microsoft Kinect depth sensors flow diagram to detect
21
depth in prototype of early development
3.4
Process in getting the skeletal point from the depth sensors
22
3.5
Relationship of platform used to develop Catch Fish game
23
3.6
The classification of classes that created for the Catch Fish
24
game in this research
3.7
Classes interaction and data flow during gameplay
26
3.8
Boat Game in early development
27
4.1
A player playing the Catch Fish game
30
xi
4.2
3-D viewer of Kinect 3-D depth data are extracted during
30
the gameplay
4.3
2-D Depth data and skeletal point mapped for each players
31
(patients)
4.4
Skeletal point data that chosen inside the game
32
4.5
Head skeletal point data of one sample player
32
4.6
Spine skeletal point data of one sample player
33
5.1
Main screen of Catch Fish game
36
xii
LIST OF TABLES
TABLE NO.
TITLE
PAGE
1
Mean and Standard Deviation for Specific Usability Scores
11
2
Mean and Standard Deviation for Overall Usability Scores
11
Classification and Comparison of Rehabilitation Serious
12
3
Games
4
Project Gantt chart for semester one
40
5
Project Gantt chart for semester two
41
xiii
LIST OF ABBREVIATIONS
VS
-
Visual Studio
CVA
-
Cerebrovascular Accidents
FAME
-
Fitness and Mobility Exercise
CNS
-
Central Nervous System
SDK
-
Software Development Kits
USB
-
Universal Serial Bus
PC
-
Personal Computer
xiv
LIST OF APPENDIX
APPENDICES
A
TITLE
List of Audio
PAGE
45
1
CHAPTER 1
INTRODUCTION
1.1
Project Background
Rehabilitation can be defined as very essential therapy to obtain back control of
muscle and joint movement. Typically rehabilitation involves method for retraining neural
pathways or training the new neural pathway in order to regain or improve the
neurocognitive functions [1]. Each year, nearly 800,000 persons are estimated to suffer
Cerebrovascular Accidents (CVA), also known as “stroke” [2]. From this figure, nearly
78 percent of survivors have impairment in executive function, attention and memory [2].
Although it has been claimed that patients can recover from stroke, there is no immediate
results as the patients need to undergo consistent and prolonged therapy.
Rehabilitation require constant discipline and willpower as it require the patients
to constantly pursue their training and therapy, even after they have managed to regain
their cognitive functions. The concept of virtual reality can simply be defined as an
interactive, computer generated environment that simulates the real world [3]. By
combining both elements together, the rehabilitation can be further improved because
virtual reality can help even severely disabled people to participate and learn safely in
simulated hazardous and complex tasks [3].
2
The key principle of stroke rehabilitation is to include a functional approach
targeted at specific activities, the frequency, and intensity of practise and to start as early
as possible within a day or weeks after stroke [4]. Conventional rehabilitation such as
FAME (Fitness and Mobility Exercise) is one of the rehabilitation programs to help stroke
patient with all possible of traditional conventional training exercise. The FAME program
has been designed to improve the mobility, fitness and fall-risk. For stroke patients, FAME
program can serve as a complement to regain healthy living [5].
Lack of interest or attention span can impair the potential effectiveness of the
therapeutic exercise. This is particularly true when a large volume of practice is needed
when dealing with many central nervous system (CNS) disorders [6]. The use of
rewarding activities has been shown to improve patient’s motivation to practice, such as
virtual reality games [7]. In an augmented reality experiment, Weghorst et. al projected
virtual objects on to the patients’ physical world to give them the impression that they
were walking over or through them, thereby restoring their mobility [8].
1.2
Problem Statement
Traditional or conventional therapy are time consuming and tedious [9]. It requires
repetitive muscle movement during training session. One such training is the balance
training where stroke patients with weak trunk strength and body balance will need to
learn shifting their weight from one leg to another or to sit on a large exercise ball and
sway their upper body. This requires determination and courage from the patients and
supervision from the therapist because many patients are afraid to move their weights
about fearing that they might fall.
Conventionally, therapist will have to closely monitor each patient and
rehabilitation sessions because people with motor disabilities experience limitations in
fine motor control, strength, and range of motion [10]. With increasing number of stroke
patients and the lack of manpower, a more efficient and effective system for rehabilitation
3
is needed to motivate people with motor disabilities to increase the number of exercises
and improve the motor proficiency and quality of life without being too dependent on
therapists. Some rehabilitation centres conduct group rehabilitations and rely on patients’
own motivation to exercise. Patients who lack motivation will feel left out and may not
practise much.
With virtual reality games, patients could continue their training because the
virtual environment provides a setting that encourage and reward movements [11]. The
properly designed game can help patients practise balancing skills in a novel way. In
addition, a competitive mode with multiplayers will allow patients to compete among
themselves and continue exercising despite the lack of supervision. A computerised
system helps the therapists to gather data during the game since patients’ movement
pattern and behaviour can be extracted from sensor readings. This data can be used to
identify specific issues with patients and to tailor their trainings based on their recovery.
1.3
Research Objective
The objectives of this research focuses on three main issues derived from the problem
statement of this project:-
i.
To determine suitable types of game for balance training in improving the trunk
strength of stroke patients for rehabilitation.
ii.
To develop a game that uses Kinect hardware and software for group
rehabilitation.
iii.
To assess and record the movement of stroke patient during gameplay for
therapist and game development improvement.
4
1.4
Research Scope
This research focuses on several scope in order to fulfil the needs and specification
of the Catch Fish game development. The first criteria needed is the fundamental research
on current studies related to rehabilitation of stroke patients, specifically focusing on
balancing training for trunk strength improvement.
Secondly, the project will use Microsoft Kinect Xbox 360 hardware and software
development kit. The software will be developed on Microsoft Visual Studio 2010
Ultimate using both XNA Software Development Kits (SDK) and XNA .NET Framework
are needed as platform to develop the algorithm of the project.
Assessment will fall into two categories, in-game assessment for the game score
and movement assessment based on skeletal data extraction from the specific game
sessions. As for in-game assessment, the player (patients) will be evaluated based on the
scoring system in which the player must collect coin or beat the current high score set in
the game.
For movement assessment based on skeletal data extraction, this assessment
architecture has been designed to work in background while the game is on-going. The
points of interest are the head and spinal skeletal point which can be used therapist to
assess a patient’s movement and track their range of motion.
5
CHAPTER 2
LITERATURE REVIEW
2.1
Conventional Rehabilitation Therapy
Rehabilitation is training which requires discipline and consistent training to
recover from stroke. With proper training and continuous practice, stroke patients may
regain control of their movements, specifically repetitive trunk and body balance while
standing or sitting. It is possible to improve outcomes related to physical activity by
implementing exercise programs carried out at home or in groups [12]. Conventional
balance training involves repetitive muscle movements. Unassisted repetitive movement
is effective in persons who have the ability to complete at least a portion of upper limbs
movements, but external assistance is required in more severely impaired patients [13].
Stroke patient who suffer balance difficulties will feel fear of falling when they
stand from sitting position. Apart from that, stroke patients will also face problem in
performing daily activities. This eventually will affect their confidence in dealing with
other people around them. With the introduction of balancing game by groups of
researchers, rehabilitation has become more comprehensive in helping stroke patients to
undergo balance training effectively. Rehabilitation should include the element of
6
excitement to keep stroke patient motivated in their training and exercise. Several
balancing games are reviewed in this chapter.
2.2
FAME (Fitness and Mobility Exercise)
The FAME (Fitness and Mobility Exercise) is one form of rehabilitation program
to help stroke patient use traditional conventional training exercise. The FAME program
has been designed to improve the mobility, fitness and fall-risk. For stroke patients, FAME
program can serve as a rehabilitation routine to regain healthy living. The objective of
FAME is to aid people who suffer stroke to regain physical activities and to minimize
complication such as falls due to their current conditions.
FAME program focuses on group activities as it considers group activity to be
important to enhance stroke patient’s adherence to the program. Interaction between two
people in group training is much more beneficial as compared to self-training because
communication and feedback occur at the same time. Thus, group training has proven to
be more effective in giving confidence and motivation for stroke patients.
Stroke is considered as one of the top risk factors for incurring fractures as a result
of a fall, with a 3 to 7 fold increase in fracture risk following a stroke. Fractures may be
caused by various factors, but improving balance and bone density can significantly stroke
patients’ risk of falls and other form fractures [14]. FAME program was introduced and
developed in Vancouver, Canada by Janice Eng, PhD, PT/OT with invaluable assistance
from Andrew Dawson, MD, FRCP, Daniel Marigold, PhD and Marco Pang, PhD, PT.
There were three trials conducted with chronic stroke patient that have shown
positive effect as compare to conventional rehabilitation therapy with specific duration of
program for each trials. The FAME Program was developed for group training program
due to the needs of the stroke patients. Stroke patients believe the group training was more
motivating as well as socially stimulating.
7
The FAME Program utilizes five basic component in their training program.
Firstly, the training program started with a short warm-up. Secondly, the training followed
up with stretching exercises to relax the specific muscle joint. Third component created to
address the muscle weakness through functional strengthening. As for the fourth
component, the training focuses on the agility and fitness with rapid repetitive movements
and finally ended with challenges balance with slower pace exercises.
One of the trials was conducted for 5 months and results showed that the control
group had bone density lost as compared to the FAME group. Apart from that, FAME
group had 30% reduction of falls in participants who had fallen prior to the trial as
compared to the control group which practices slower moving program with weightbearing and stretching exercises.
In order to further assess the patients, the FAME group were also screened by a
standard cardiovascular stress test which permitted a higher intensity of training compared
to previous trials and control group. This positive outcome shows the improvement of the
FAME Program in all aspect which include walking endurance, mobility, and fall risk
improvement over all trials assessed by different FAME groups.
8
2.3
Neverball: Virtual Environment Game-Based Balance Exercise
Neverball is an open source computer game which has been design for player to
tilt the on-screen floor to roll a ball through an obstacle course and collect virtual coins
before time runs out as shown in Figure 2.3.1 [15].
Figure 2.3.1: Neverball in-game interface and gameplay.
Figure 2.3.2 shows the Neverball game set up, comprising off-the shelf
components such as wobble board, orientation tracker, an USB communication cable and
a personal computer. To set the wobble board as input controller for the system, the
orientation tracker is attached to the surface of the wobble board. This inertial-based
orientation sensor allows 3 degree of freedom detection, therefore tracking the 3D
orientation of the wobble board using the balance exercise setup as shown in Figure 2.3.2
[15].
9
Figure 2.3.2: Neverball played on a wobble board.
The result demonstrated a positive outcome towards virtual balance training
usability. Twelve participants (six female and six male) consisting of stroke patients tested
balance training set-up. Likert scale-type questions are divided into 10 separate usability
categories to get feedback on the functionality, user input, system output (display), user
guidance, consistency, flexibility, simulation fidelity, error correction/handling and
robustness, sense of immersion/presence, and overall system usability. Likert scales were
widely used in survey studies for attitude measuring [16]. These outcome were tabulated
and were used to analyse the functionality of the system. Neverball game showed that
virtual environment game-based has helped to improve rehabilitation training as it aided
stroke patients to undergo repetitive training without noticing that the time passed.
10
Table 1: Mean and standard deviation for specific usability scores
Usability Category
Max*
Min+
Score ± SD
Functionality
35
7
29.67 ± 2.77
User Input
70
14
54.25 ± 5.01
System Output
100
20
83.67 ± 6.08
Consistency
40
8
34.92 ± 3.09
Flexibility
30
6
22.67 ± 2.42
Simulation Fidelity
55
11
45.25 ± 6.18
Error Correction
35
7
24.67 ± 3.23
Sense of Immersion
50
10
37.75 ± 5.53
Overall System
55
11
48.00 ± 3.49
Usability
*Figure indicates the maximum possible score for individual categories
+Figure indicates the minimum possible score for individual categories
Table 2: Mean and standard deviation for overall usability scores
Usability Category
Max*
Min+
Score ± SD
Functionality
5
1
4.42 ± 0.67
User Input
5
1
4.25 ± 0.87
System Output
5
1
4.75 ± 0.62
Consistency
5
1
4.75 ± 0.62
Flexibility
5
1
4.17 ± 0.83
Simulation Fidelity
5
1
4.25 ± 0.87
Error Correction
5
1
4.42 ± 0.79
Sense of Immersion
5
1
4.08 ± 0.90
Overall System
5
1
4.83 ± 0.58
Usability
*Figure indicates the maximum possible score for individual categories
+Figure indicates the minimum possible score for individual categories
11
2.4
Serious Games for Rehabilitation
Serious Game aims for problem solving in diverse areas and evaluate the potential of the
games [1]. Although the term game agreed by different author that refer to the use of
computer simulation sorely for entertainment, but virtual reality based game can offer
more benefits depending on the content and purpose of the game [1].
Serious Games have been applied in different areas such as corporate and military
training [17], health [18] – [20], education [21] – [25] and others [26]. These areas are
taken into consideration as it relates with other division such as e-learning, eduitament
and game-based learning. For example, in rehabilitation the requirement of timing
management, patient’s selection, choice of rehabilitation program and other is important
because rehabilitation procedure itself must be specific for specific stroke patients
training.
Table 3 shows previous research that have used the concept of Serious Games for
Rehabilitation. One of the research by Betker et. al used body weight movement as the
interaction or input to the game. This game used the concept of balance training as
compared to the other games listed in Table 3. This game was convenient as patients were
able to perform the training at home [27].
Table 3: Classification & Comparison of Rehabilitation Serious Games [1].
12
13
2.5
Balloon Burst
Another game, the Balloon Burst game required player to control the cursor to
move it over the balloon to pop it. The assessment is measured by the total number of
balloons, the number of balloons popped, and the movement range of the player. The
concept of virtual reality such Balloon Burst game deliver player the excitement to
conduct training by themselves and able to progress further than the normal conventional
training procedure.
Figure 2.5.1: System setup for game based exercises for dynamic short-sitting balance
rehabilitation [27].
The game setting is as shown in Figure 2.5.1, where patient sat on the pressure mat
(1), which was connected to the laptop by the interface box (2). The laptop displayed the
game Balloon Burst. The pressure mat was placed on top of the SwisDisk (3); the Physio
Gymnic (4) ball also was depicted [27]. Balloon Burst deliver effective balance training
to stroke patient to assess themselves and proven to helpful for the therapist to monitor
their progression from time to time.
14
2.6
PlayMancer: A Serious Gaming 3D Environment
PlayMancer aims to implement a framework and a platform for serious game by
augmenting existing 3D gaming engines [29]. One of the important objectives is to
evaluate the proposed framework and gaming infrastructure by developing and testing
series of serious game modules as applied to two application domain. These two
application domain were physical rehabilitation, and therapeutic support and lifestyle
management programs for behavioural and addictive disorders [29].
Based on Figure 2.6.1, PlayMancer platform was composed of three major parts
which is Game Engine, Game or Application Manager, and Hardware Abstraction Layer
[29]. The Hardware Abstraction Layer provide interface to the input and output devices
and auxiliary component. During runtime, profiles is updated and models are generated
according to the preferences for both single player and multiplayer environment.
PlayMancer platform was used to overcome barriers by augmenting existing 3D
engines with new possibilities and thusly creating a development framework for serious
games [29]. In future game development can further improve older gaming engines by
bringing new architecture which enable more assessment and results.
Figure 2.6.1: Simplified block diagram of PlayMancer platform [29].
15
16
2.7
Rehabilitation Gaming System (RGS)
Rehabilitation Gaming System (RGS) is virtual reality based game introduced for
enhancement of functional recovery after lesions to the nervous system using non-invasive
multi-modal stimulation [31]. With virtual reality, training protocols can be well
controlled within specifically defined interactive scenarios [31]. In given condition, RGS
is capable to generate task-specific training scenarios which designed for the rehabilitation
of upper limbs. The training can be monitored and quantified as for the improvement of
patients over time.
Figure 2.7.1: The Rehabilitation Gaming System set-up [31].
Game set-up is shown in Figure 2.7.1, where patient resting his or her arms on the
table surface while facing the display. Movement of the arms are visually captured by
camera that placed on top of the main display. Movement were captured from the colour
patches which located at wrists and elbows. A pair of data gloves were also equipped to
the patients to measure finger flexure. The main display were used to show the avatar
moving according to the movements of patients performs task in virtual reality.
17
Figure 2.7.2: Virtual player’s arm tried to “hit” the flying sphere [31].
Based on Figure 2.7.2, in-game training requires patients to intercept spheres that
fly towards him or her by hitting those sphere using virtual arms displayed on the main
display. The purpose was to allow patients to gain their upper limbs control. As the
patients continue to assess themselves, the parameter of the game continue to adapt to
keep the performance level at around 70% [31].
18
2.8
Catch Fish: Virtual Balancing Game to Enhance Trunk Strength for Stroke
Patients
Catch Fish game overcome the limitation of both conventional and modern
rehabilitation therapy by using the virtual reality. Derived from the FAME Program, Catch
Fish was designed for group training because the group training is important factor in
improving rehabilitation training, specifically balance training in order to improve their
trunk strength. Based on Figure 2.8.1, Catch Fish game allowed single player mode and
multiplayer mode. The design used the Microsoft Kinect hardware with its SDK and XNA
as development platform to increase the robustness and smoothness of the gameplay
Figure 2.8.1: Catch Fish game menu.
19
CHAPTER 3
RESEARCH METHODOLOGY
3.1
Hardware
In this chapter, the hardware and software development of Catch Fish game are
discussed. Based on Figure 3.1, the Kinect consists of several important components.
First is 3-D depth sensor, which tracks three-dimensional movement of player’s body
within specified region. Secondly, an RGB camera which reads red, green, and blue colour
array in distinguishing specific outcome such as getting distance of an object or takes ingame pictures and videos. Lastly, the multiple microphone array is used for speech
recognition and chat programs.
Figure 3.1: Kinect sensors (1) Infra-red (IR) Depth Sensors, (2) IR Camera, (3)
Microphone array, (4) Tilt sensor and motor.
20
This project only requires the function of the 3-D depth sensor, which will capture
the player’s joints or specifically the skeletal point data and deliver to the system algorithm
for further processing. Since the project primarily focuses on trunk movements, therefore
the skeletal points of interest would be the human spinal area. The depth sensor captured
the skeletal points and directly sent the data to be represented as movements on screen for
further analysis and processing.
Microsoft uses different alternatives in measuring the depth of object in space.
Older software programs used IR-depth sensors to differentiate the colour and texture to
distinguish objects from their backgrounds. Conversely, Microsoft uses the camera to
transmit invisible near-infrared light and measure its “time-of-flight” after it reflects off
by the objects. The difference was the processing speed. In Kinect, the “time-of-flight” is
much faster to be calculated as compared to measuring colour and texture differences.
Complexity increases if the colour and texture at background is same as objects within the
area.
Figure 3.2: Microsoft Kinect camera and microphone used to measure object’s depth.
21
Start
1.
2.
Invisible light source illuminates the subject
Sensors chip measures the distances of the light
travels, to each pixel within the chip.
3.
Unique embedded imaging software uses the “depth
map” to perceive and identify objects in real time
4.
End-user device react appropriately to algorithm that
have designed.
End
Figure 3.3: Microsoft Kinect depth sensors flow diagram to detect depth in prototype of
early development.
Based on both Figure 3.2 and Figure 3.3, the camera acts as sensor in measuring
the time-of-flight which theoretically work like sonar. The concept works by measuring
how long the light takes to return. Besides that, infrared modelling also partially solves
the problem of ambient light from the older modelling. This is because the sensor is not
designed to register visible light. Therefore, it does not generate as much false positive in
rendering the depth of an object detected. For the skeletal point detection, Microsoft uses
maps or draws skeletal point on the player’s body through its program and display it.
Figure 3.4 shows the flow chart depth and skeletal point detection using the Kinect sensor.
22
Start
IR-Depth sensor transmit light into the
field
IR-Camera sensor capture the
reflected light
Draw the object
Kinect Software measure the
depth on the main
time-of-flight
display
Software send the
data to the library
Mapped the depth data
into algorithm
Library identified and process the
algorithm into skeletal point
Joint the skeletal point to form
player’s body on main display
End
Figure 3.4: Process in getting the skeletal point from the depth sensors.
23
For this project, the Kinect depth sensor was first initialized to detect nearest
player’s body from the sensor. Next, the location of player’s joints were extracted to form
XY plane of movements. Finally, quantization error reduction was performed to smoothen
movement data for tracking. These skeletal data were sent into the algorithm for further
processing. Player was able to control their screen movements based on the changes
captured by the Kinect’s depth sensor.
3.3
Software
Based on Figure 3.5, several software platforms were used to construct the
algorithms for this game. Microsoft Visual Studio 2010 Ultimate (VS) was chosen as the
primary platform to develop the Catch Fish game. This was because VS platform was
compatible with XNA game development plug-in (pre-defined library) and its frameworks
(XNA Frameworks) for game design and algorithm support. The Catch Fish game was
developed using C# language.
Kinect sensors
initialization
Kinect libraries
provided by SDK
Windows and Xbox
game development
project integration
Figure 3.5: Relationship of platform used to develop Catch Fish game.
24
To access and control the Kinect, the Microsoft Kinect SDK (Software
Development Kit) is needed. This project used version 1.8 of the Microsoft Kinect SDK
which was downloadable from Microsoft website. Using these libraries, programmers
would be able to interface with the Kinect hardware such as the IR Depth sensors.
Other than that, a custom library was also used to correctly map the graphical items
within the algorithm. Class objects such as Sprites.cs and Object.cs were used to ensure
the image or object that were designed can be easily controlled within the game algorithm.
GestureRecognizationEngine.cs, GestureBase.cs, and GestureAngleCalc.cs were derived
and modified from “Durian Runtuh” game developed by Yong Bang Xiang for the full
control of Kinect inputs to the Catch Fish game. These classes interacted with each other
for the full function of the main program. Figure 3.6 below shows the relationship of all
custom classes to the main class which is Game1.cs.
Object.cs
Object
control and
behaviour
Sprites.cs
Game1.cs
Kinect process as
primary input to
the game
GestureRecognization
Engine.cs
GestureBase.cs
GestureAngleCalc.cs
Figure 3.6: The classification of classes that created for the Catch Fish game in this
project.
25
As shown in Figure 3.6, Object.cs is used to control the game logic such as
collision of objects that affect the scores of player and the object size as well as initial
position in the game. Sprites.cs is used to control the animation, texture selection and
update behaviour of the object created in the game. As for the Kinect processing part,
GestureBase.cs takes the initial, current and final updates on skeletal track on the player.
If fails to detect an updates, the value will return “false” to the main algorithm.
For the GestureAngleCalc.cs, most part of game calculation were taken from the
update states of the player’s movement. For example, the initial position were recorded
and would be compared to the current position which were tracked in every single frame.
As for the GestureRecognizeEngine.cs, it helped the game to update the availability of the
player in the game and returned it to the main algorithm. Thus, it will help to reduce
unwanted process if skeletal point cannot be traced due to absence of player.
Figure 3.7 shows the algorithm flow of Catch Fish game in every frame. In each
frame, the Kinect sensors will update the data obtained from the software libraries and
would be process in each classes to perform specific task. This is because the Kinect
sensors need simulation from its own libraries to perform skeletal mapping on the player
in each frame count
26
Start
Kinect initialization, game logic
initialization.
GestureRecognizeEngine scan for player
available player
No
Player?
Yes
Send data to GstureBase to update the
initial, current and final position of player
GestureAngleCalc will perform
calculation by comparing values taken
from previous frame and the next frame
Object.cs is use to determine objects size
and initial position.
Value retrieved from GestureAngleCalc is
use to perform process to main algorithm
by Sprites
End
Figure 3.7: Classes interaction and data flow during gameplay.
27
3.4
Boat Game
Initially, Boat Game was designed for this project using Microsoft Kinect Libraries
as main algorithm for the game. Boat Game was derived from the ShapeGame which is
available in the Microsoft Kinect SDK. Microsoft Visual Studio 2010 Ultimate was
chosen to be the only platform for the game development and modification.
The concept was very simple. Players (stroke patients) need to move their body in
play area to collect points by collecting colourful circle released from top of the screen.
Every time players moved their upper body either to the left or to the right, the object boat
on the screen will follow their movement.
Figure 3.8: Boat Game in early development.
Based on the Figure 3.8, in early development of Boat Game, the only focus was
to track player’s skeletal point, specifically the head and spinal point on the player’s body.
The attempt was successful and the Boat Game was able to detect up to two active players
at one time. However, after the assessment algorithm had been integrated with the Boat
Game, several bugs were identified. One of the bug was related to the assessment log files
where the text file “log.txt” was successfully created, but the content was empty.
Other than that, movement of the object that represented player in the game was
not suitable for the stroke patients. The scale factor applied in the algorithm was not
28
working properly and movement distortion is very large. Patients will find it difficult to
move their body based on movement of boat represented on the screen. Hence, the
development of Boat Game was discontinued and replaced with Catch Fish game.
Using the previous Visual Studio platform, the Catch Fish game was created. This
project used an available commercial sensor, which is the Microsoft Kinect to detect
movements. For the software algorithm, the project integrated several platform and used
both pre-defined and custom written libraries.
29
CHAPTER 4
RESULTS AND DISCUSSION
4.1
Game Logic and In-Game Assessment
The Catch Fish game (Figure 4.1) was successfully developed to train trunk
strength for stoke patients through swaying movements. Players (patients) were required
to sway their body to move a boat left and right within the game. The boat was used to
collect fishes that drop from the sky and to dodge stones to avoid being penalized. To pass
one game level, players have to collect 150 points.
The fish acted as an in-game object for them to catch and players gained points by
using their boats which was the representative of their own movements mapped into the
virtual game itself. Stone caused point deduction and number of lives deduction which
helped to increase challenges for the players.
30
Figure 4.1: A player playing the Catch Fish game.
An algorithm was written to extract 3-D skeletal points to analyse the movement
patterns of the player. The 3D data was used to record the full movements of the player
and was extracted when players were playing the game. Figure 4.2 shows the 3D Kinect
depth data extracted during game play. From the 3D data, the XY values were extracted
to be mapped onto the screen for 2D object movements on screen, as shown in Figure 4.3.
Figure 4.2: 3-D viewer of Kinect 3-D depth data are extracted during the gameplay.
31
Figure 4.3: 2-D Depth data and skeletal point mapped for each players (patients)
Specific skeletal points used inside the game algorithm were the head skeletal
point and spinal skeletal point (Figure 4.4). Changes of these two points will cause the
boat inside the game to be moved to the left or to the right.
Figure 4.4: Skeletal point data that used inside the game.
32
4.2
Analysis and Data Extraction
The skeletal data were tracked for each frame with speed of 30 frames per second.
The data was then tabulated for analysis. Graph representation of the data can be seen in
Figure 4.5 and 4.6 where the movements were plotted based on pixel position on screen
(y-axis) versus frame number (x-axis). The data extracted from the graph is important to
study the pattern and behavior of the players (patients) at a particular session.
Pixel position
on screen
Head Skeletal Point Data
900
800
X - Axis
Y - Axis
700
600
500
400
300
200
100
1
25
48
71
94
117
140
163
186
209
232
255
278
301
324
347
370
393
416
439
462
485
508
531
554
0
Frame
Number
Figure 4.5: Head skeletal point data of one sample player.
33
Pixel position
on screen
800
Spine Skeletal Point Data
X - Axis
Y - Axis
700
600
500
400
300
200
100
1
23
45
67
89
111
133
155
177
199
221
243
265
287
309
331
353
375
397
419
441
463
485
507
0
Frame
Number
Figure 4.6: Spine skeletal point data of one sample player.
34
CHAPTER 5
CONCLUSION AND FUTURE RECOMMENDATIONS
5.1
Introduction
Catch Fish game could be a motivating and interactive game to help therapists in
conducting rehabilitation training sessions. The game allows therapists to obtain specific
data that can be used to improve rehabilitation for the stroke patients. The more motivated
the patients are, the more training they would voluntarily perform and the higher the
intensity they will put into their movements, which may translate into more effective
training. The scoring system implemented in-game was helpful in setting up goals for the
patients for continued training.
5.2
Catch Fish Game
Based on the research objectives, Catch Fish game has been successfully designed
to complement conventional rehabilitation. This game involved balancing exercise for
trunk strength of stroke patients. The Catch Fish game also has been redesigned from
totally new algorithm after the discontinued work of Boat Game, discussed earlier in this
35
thesis. Catch Fish provided an alternative for stroke patients to undergo their training more
convinently and interactively.
This research has overcome the limitation of hardware technologies in Neverball
and Serious Game by using only Microsoft Kinect hardware and software to develop the
Catch Fish game. Kinect hardware is commercially available to the end-user and allows
user to fully utilize the software resources available in the given libraries or customize it
accordingly to their application.
Catch Fish game successfully integrated multiplayer mode to help stroke patient
to motivate each other through healthy competition. Self-confidence and excitement will
make stroke patients to continue their balance training for longer duration using Catch
Fish game.
5.3
Limitations
The early version of Microsoft Kinect was designed separately for both Xbox 360
edition and PC edition. Some of the features available in the Xbox version are not
available for PC version. However, as for Kinect for Windows, useful near-tracking mode
are available for this version and includes a license for commercial distribution of apps.
For the Xbox version, it is only available for developers to develop their game or software
that runs smoothly on Xbox 360 console.
Apart from that, XNA game development allow publication for only PC and Xbox
360 console. The support for XNA are no longer available in the latest edition of Visual
Studio, such as Visual Studio 2012 edition. Not only the supports for XNA has been
discontinued, XNA game development also require the latest edition of .NET framework
in order to work and run properly.
36
5.4
Recommendations
With latest release of Microsoft Kinect version 2.0, Microsoft has successfully
improved the weaknesses found on previous versions of Kinect. The improvement of
latest Microsoft Kinect allows developers to trace and program their algorithm more easily
as the depth sensor in Kinect version 2.0 is more accurate and detailed. This can further
help the developers to test their algorithm that requires very precise detection using the
improved IR-Depth sensors.
An alternative for XNA game development, MonoGame, has become a popular
tool for Visual Studio. This is because MonoGame supports almost all platforms available
in the market today such as IOS, Android and Windows Phone. With code structure almost
identical to XNA game development, MonoGame can help developer to code their ideas
on different operating systems for different markets and target groups.
Figure 5.1: Main screen of Catch Fish game
Figure 5.1 shows the home screen of the Catch Fish game. High score and multiple
levels of play will provide motivation for players to give their best during training. To set
proper targets suitable for stroke patients’ capability, a clinical study will have to be
conducted to assess the average score for various stages of stroke. This will make the
game more achievable yet, not too simple for the patients.
37
CHAPTER 6
PROJECT MANAGEMENT
6.1
Introduction
Project management is crucial in order to achieve all project goals, in this case, the
objectives of the study. Project management is divided into project planning, organizing
and controlling the resources within a time interval. In this study, there are some
constraints and limitations that need to be overcome by the researcher, which are research
scopes, research time, research budget and human resource to perform the research
activity.
6.2
Project Schedule
Gantt charts for Semester one and two are shown in Table 4 and 5 respectively.
From the Gantt chart in Table 4, there is some delay in the project background study. This
is due to the late assignment of supervisor to the students. Other than that, the 4th year
problem-based laboratory started earlier than the date of the announcement of supervisor
in charge for the final year project. After discussing with the supervisor on the title that
need to be done, then only the study on the previous works were done. Other than that,
38
other tasks commenced as proposed on date. The red cells represent the expected duration
of each task, while the green cells represent the actual duration of each task. Meanwhile,
Table 5 shows the project Gantt chart for semester two. In contrast to semester one, there
was an unexpected long delay in some task due to the long simulation time.
Table 4: Project Gantt chart for semester one
Activity
1
Research on
Kinect
Software
preparation
Online
Tutorial
Hand drawn
UI design
UI design
using Visual
Studio 2010
Working with
XNA
Boat Game
Complete and
Report writing
Report
submission
September
2 3 4
2013
October
November
December
5 6 7 8 9 10 11 12 13 14 15 16
39
Table 5: Project Gantt chart for semester two
Activity
Implement
multiplayer
versions in the
game
Analysis of
game
algorithm to
improve
smoothness of
the game
Discontinued
Boat Game
and proceed to
new game
development
(Catch Fish)
Revised C#
and XNA
coding
Objects and
Sprites design
Kinect as main
input with
multi-player
integration and
skeletal data
extraction
Analysis the
skeletal
tracking data
from log files
Project
completed and
tested (Second
Stage) and
Thesis writing
February
March
2014
April
May
June
40
REFERENCES
1. Rego, Paula, et al. "Serious games for rehabilitation: A survey and a classification
towards a taxonomy." Information Systems and Technologies (CISTI), 2010 5th
Iberian Conference on. IEEE, 2010.
2. Rand, Debbie, et al. "Feasibility of a 6-month exercise and recreation program to
improve executive functioning and memory in individuals with chronic stroke."
Neurorehabilitation and neural repair (2010).
3. Andreae, Michael Henning. "Virtual reality in rehabilitation." BMJ: British
Medical Journal 312.7022 (1996): 4.
4. Hsieh, Ru-Lan, et al. "Additional therapeutic effects of electro acupuncture in
conjunction with conventional rehabilitation for patients with first-ever ischemic
stroke." Deutsche Zeitschrift fuer Akupunktur51.1 (2008): 56-57.
5. Pang, Marco, et al. "A Community‐Based Fitness and Mobility Exercise Program
for Older Adults with Chronic Stroke: A Randomized, Controlled Trial." Journal
of the American Geriatrics Society 53.10 (2005): 1667-1674.
6. Betker, Aimee, et al. "Video game–based exercises for balance rehabilitation: a
single-subject design." Archives of physical medicine and rehabilitation 87.8
(2006): 1141-1149.
41
7. Cogan, Madey, et al. "Pong game as a rehabilitation device." Fourth Annual
Conference on Systems and Devices for the Disabled. Seattle, Wash: University of
Washington School of Medicine. 1977.
8. Weghorst,
Suzanne.
"Augmented
reality
and
Parkinson's
disease."
Communications of the ACM 40.8 (1997): 47-48.
9. Levin,
Mindy.
"Can
virtual
reality offer
enriched
environments
for
rehabilitation?" Expert Review of Neurotherapeutics 11.2 (2011): 153-155.
10. Chang, Yao-Jen, et al. "A Kinect-based system for physical rehabilitation: A pilot
study for young adults with motor disabilities." Research in developmental
disabilities 32.6 (2011): 2566-2570..
11. Fulk, George. "Locomotor Training and Virtual Reality‐based Balance Training
for an Individual with Multiple Sclerosis: A Case Report." Journal of Neurologic
Physical Therapy 29.1 (2005): 34-42.
12. Roos, Ewa. “Effectiveness and practice variation of rehabilitation after joint
replacement.” Current opinion in rheumatology 15.2 (2003): 160-162.
13. Lum, Peter, et al. “Robot-assisted movement training compared with conventional
therapy techniques for the rehabilitation of upper-limb motor function after
stroke.” Archives of physical medicine and rehabilitation 83.7 (2002): 952-959.
14. Marigold, Daniel, et al. "Exercise leads to faster postural reflexes, improved
balance and mobility, and fewer falls in older persons with chronic stroke."
Journal of the American Geriatrics Society 53.3 (2005): 416-423.
15. Fitzgerald, Diarmaid, et al. "Development and user evaluation of a virtual
rehabilitation system for wobble board balance training." Engineering in Medicine
42
and Biology Society, 2008. EMBS 2008. 30th Annual International Conference of
the IEEE. IEEE, 2008.
16. Göb, Rainer, et al. “Ordinal methodology in the analysis of Likert scales.” Quality
& Quantity 41.5 (2007): 601-626.
17. Numrich, Susan. “Culture, Models, and Games: Incorporating Warfare’s Human
Dimension.” IEEE Intelligent Systems 23.4 (2008): 0058-61.
18. Blackman, Sue. “Serious games… and less!.” ACM Siggraph Computer
Graphics 39.1 (2005): 12-16.
19. Macedonia, Mike. “Virtual worlds: a new reality for treating post-traumatic stress
disorder.” Computer Graphics and Applications, IEEE 29.1 (2009): 86-88.
20. Von, Christiane, et al. "To Game or Not to Game?" IEEE Software (2009): 92-94.
21. Mayo, Merrilea. “Games for science and engineering education.” Communications
of the ACM 50.7 (2007): 30-35.
22. Kelly, Henry, et al. “How to build serious games.” Communications of the
ACM50.7 (2007): 44-49.
23. Zyda, Michael. “Creating a science of games.” COMMUNICATIONS-ACM 50.7
(2007): 26.
24. Westera, Wim, et al. “Serious games for higher education: a framework for
reducing design complexity.” Journal of Computer Assisted Learning 24.5 (2008):
420-432.
43
25. Connolly, Thomas, et al. “An application of games‐based learning within software
engineering.” British Journal of Educational Technology 38.3 (2007): 416-428.
26. Zielke, Marjorie, et al. “Serious games for immersive cultural training: creating a
living world.” IEEE computer graphics and applications 29.2 (2009): 49-60.
27. Betker, Aimee, et al. “Game-based exercises for dynamic short-sitting balance
rehabilitation of people with chronic spinal cord and traumatic brain
injuries.” Physical therapy 87.10 (2007): 1389-1398.
28. Ma, Minhua, et al. “Serious games for movement therapy after stroke.” Systems,
Man and Cybernetics, 2008. SMC 2008. IEEE International Conference on. IEEE,
2008.
29. Conconi, Alex, et al. “Playmancer: A serious gaming 3d environment.” Automated
solutions for Cross Media Content and Multi-channel Distribution, 2008.
AXMEDIS’08. International Conference on. IEEE, 2008.
30. Caglio, Marcella, et al. “Video game play changes spatial and verbal memory:
rehabilitation of a single case with traumatic brain injury.” Cognitive processing
10 (2009): 195-197.
31. Cameirão, Mónica, et al. “The rehabilitation gaming system: a review.” Stud
Health Technol Inform 145 (2009): 65-83.
32. Cameirão, Mónica, et al. "A virtual reality system for motor and cognitive
neurorehabilitation." Challenges Assistive Technol 20 (2007): 393-397.
33. Ryan, M., et al. “Rehabilitation Games: Designing Computer Games for Balance
Rehabilitation in the Elderly.” (2012).
44
34. Schuster.
"Rehabilitation
of
TBI
MEDICOPHYSICA 38.1 (2002): 39-44.
using
RehaCom."
EUROPA
45
APPENDIX A
LIST OF AUDIO
1. SoundTeMP, Theme of Aldebaran, Ragnarok Online BGM
2. SEGA, City Escape, Sonic Generation OST