Mobile PC Development

Transcription

.N
ET
Developing Windows Vista™ applications for notebooks, Tablet PCs, and UMPCs
2007 Volume 4 / Issue 2
uild Ultra-Mobile
B
Applications
Windows Vista Mobility
evelop for
D
Windows
SideShow
WPF on
Mobile PCs
w w w. c o d e - m a g a z i n e . c o m
US $ 5.95 Can $ 8.95
table of contents
Features
6Being Ultra-Mobile
57Inking in ASP.NET 2.0, AJAX, and IE7
Markus ponders how developers need to plan applications
that allow for different hardware and software minimums than
developers have enjoyed the past few years.
What changes can developers make to their Web-based
applications to make them easier for users who use ink
rather than a traditional keyboard? Julie explains.
Markus Egger
Julie Lerman
8Welcome from the Mobile and
Tailored Platforms Group
66Basics of Ink in
Windows Presentation Foundation
Frank reviews some of the “big picture” improvements that
the Windows Vista™ operating system will bring to mobile
computing.
Microsoft baked support for ink right into Windows
Presentation Foundation (WPF). Billy reveals how you get
started using ink and WPF.
Frank Gocinski
Billy Hollis
12How to Be Where Your Customer Wants to Be
Tara and Frank discuss some of the expectations and opportunities that mobile developers will have to meet the demands of
increasingly mobile customers.
Tara Prakriya and Frank Gocinski
15Developing Windows Vista Applications
for the Ultra-Mobile PC
Todd, a member of the mobile development team at Microsoft,
describes some of the design goals of the Ultra-Mobile PC.
Todd Landstad
29Windows Vista Mobility: Why Should You Care?
Mobility pundit ActiveNick spells out what design improvements make Windows Vista a compelling mobile platform for
developers to target.
Nickolas Landry
38The Proper Developer Environments
for Mobile PC, Tablet PC,
and Ultra-Mobile PC Applications
Eliot explains the object models, the SDKs, the forms packages,
and other elements that developers need to set up their development environments for targeting mobile platforms that run
Microsoft Windows.
Eliot Graff
44Adapting Existing Applications
to Work on UMPCs
Often a company wants to take an application they already use
and adapt it to work on new mobile platforms. So what do you
do when the new platform isn’t much larger than a personal
DVD player? Dr. Neil breaks it down.
72Exploring Ink Analysis
The Tablet PC does an amazing job of recognizing handwriting. Windows Vista will do even better. Frank explains
how developers can help increase the chance that their inkenabled applications will correctly recognize handwriting.
Frank LaVigne
79Best Practices in Game Design
for the Ultra-Mobile PC
While the UMPC does run a full version of Windows, you
may wonder what changes to make to a game (or almost
any other application) that gives users the best experience
on the UMPC’s smaller size.
Matt Gillespie, Michael Finkel, Victoria Bailey
84SideShow Development in Depth
Microsoft is developing technology to allow small devices
to deliver information without having to boot up Windows.
Dr. Neil explores this technology and explains how you can
build an application using it.
Dr. Neil Roodyn
92Tablet PC Input Panel Programmability
Jan-Kristian spells out how developers can use the ITextInputPanel API to tell Input Panel how you want it to behave
around the controls in your application.
Jan-Kristian Markiewicz
94Exploring Tablet PC Application Deployment
OK, you’ve built that Tablet PC application that you want
to deploy to Windows XP and Windows Vista clients.
How do you do it? CoDe Magazine’s own Editor-in-Chief
walks you through the steps.
Rod Paddock
Dr. Neil Roodyn
52Introduction to Windows Touch
Technology in Windows Vista
Hilton explains why we’ll find Windows Vista a more compelling touch-driven way to perform everyday computing tasks.
Hilton Locke
Departments
5Code Compilers
98Advertisers Index
US subscriptions are US $29.99 for one year. Subscriptions outside the US pay US $44.99. Payments should be made in US dollars drawn on a US
bank. American Express, MasterCard, Visa, and Discover credit cards are accepted. Bill me option is available only for US subscriptions. Back issues
are available. For subscription information, send e-mail to [email protected] or contact customer service at 832-717-4445 ext 10.
Subscribe online at www.code-magazine.com
CoDe Component Developer Magazine (ISSN # 1547-5166) is published bimonthly by EPS Software Corporation, 6605 Cypresswood Drive., Suite
300, Spring, TX 77379 U.S.A. POSTMASTER: Send address changes to CoDe Component Developer Magazine, 6605 Cypresswood Drive., Suite
300, Spring, TX 77379 U.S.A..
Table of Contents
www.code-magazine.com
code compilers
2007
Volume 4 Issue 2
Group Publisher
Markus Egger
Associate Publisher
Rick Strahl
Managing Editor
Ellen Whitney
Content Editors
H. Kevin Fansler
Erik Ruthruff
Writers In This Issue
Victoria Bailey
Michael Finkel Frank Gocinski
Billy Hollis
Todd Landstad
Julie Lerman
Tara Prakriya
Markus Egger
Matt Gillespie
Eliot Graff
Nickolas Landry
Frank LaVigne
Hilton Locke
Rod Paddock
Dr. Neil Roodyn
Technical Reviewers
Frank Gocinski
Ellen Whitney
Art & Layout
King Laurin GmbH
[email protected]
Production
Franz Wimmer
King Laurin GmbH
39057 St. Michael/ Eppan, Italy
Printing
Fry Communications, Inc.
800 West Church Rd.
Mechanicsburg, PA 17055
Advertising Sales
Vice President, Sales and Marketing
Tom Buckley
832-717-4445 ext 34
[email protected]
Sales Managers
Erna Egger
+43 (664) 151 0861
[email protected]
Tammy Ferguson
832-717-4445 ext 26
[email protected]
Circulation & Distribution
General Circulation:
EPS Software Corp.
Newsstand:
Ingram Periodicals, Inc.
Media Solutions
Source Interlink International
Subscriptions
Circulation Manager
Cleo Gaither
832-717-4445 ext 10
[email protected]
US subscriptions are US $29.99 for one year.
Subscriptions outside the US are US $44.99.
Payments should be made in US dollars drawn
on a US bank. American Express, MasterCard,
Visa, and Discover credit cards accepted. Bill
me option is available only for US subscriptions.
Back issues are available. For subscription
information, email [email protected]
or contact customer service at 832-717-4445 ext 10.
Subscribe online at
CoDe Component Developer Magazine
EPS Software Corporation / Publishing Division
6605 Cypresswood Drive, Ste 300,
Spring, Texas 77379
Phone: 832-717-4445
Fax:
832-717-4460
TextTextTextTextTextTextTextTextTextTextTextText
Editorial
Being Ultra-Mobile
H
ow often have you been at user group meetings, conferences, or other geek-gatherings
discussing the “good old days”? The days
when we all were true code-warriors, feeding
punch-cards into ancient readers, or replacing
tubes and removing literal bugs from our hardware. We have come a long way since the good
old days, when computers filled entire rooms.
Those were the days when computers were
about as stationary as it gets, short of a bunker
or Hoover dam. “Being mobile” meant carrying
punch cards from one room to another.
Today, things are quite a bit different. Mobile
computing has become the most pervasive form
of computing. Notebook computers outsell
desktops. As new portable computers rival their
bulky counterparts in performance, even hardcore users such as gamers make the move to mobility. More often than not, a laptop computer
replaces the desktop machine in today’s offices
and living rooms, making the portable computer
not just a companion device, but the user’s main
computer.
Companion devices are important. Personally, I
have fallen in love with my new Ultra-Mobile PC
(UMPC). In fact, I am writing this very article on
a plane using my UMPC. I am not ready to give
up my bigger Tablet PC yet, but it sure is nice to
have a device that is small, less than two pounds
in weight, has an external (optional) keyboard,
and runs all my Microsoft® Windows® applications. I was using my Tablet PC earlier on this
flight, but ran out of battery after about three
hours. My UMPC, on the other hand, runs for
roughly seven hours on one battery and I have
two batteries, allowing me to work for more than
an entire work-day without the need to recharge.
I have considered buying a slate Tablet PC in
addition to my convertible Tablet PC for those
times when I just want to lay on the couch and
browse the Web or read a digital article or book.
But UMPCs are a lower-cost alternative.
From a hardware point of view, the world is
great for mobile PCs, no matter whether they
are notebooks, Tablet PCs, or Ultra-Mobile PCs.
And clearly, for software vendors and developers, this is the largest and most mainstream
market to go after these days. However, I am
TextTextTextTextTextTextTextTextTextTextTextText
Editorial
always surprised how few developers ever give
any special thought to mobile PC development.
“After all,” so the reasoning seems to go, “mobile
PCs are just regular Microsoft Windows PCs. So
there is nothing special to do as a developer…”.
But I beg to differ! Mobility means a lot more
than carrying your laptop from your office to the
conference room. Mobility means that you can
use a mobile PC anywhere and at any time, no
matter whether the person is holding the PC in
their hand, balances it on their lap, or has it on
a desk at the office. They do not run out of battery when they are needed most (such as at the
end of a meeting when that final decision needs
to be made). They still provide access to that
important piece of information, even when the
connection to the database has been lost. And
mobile PCs allow a user to search for a piece of
information effectively, even if they don’t have
a keyboard.
These features do not come for free of course.
Microsoft Outlook® doesn’t have offline capabilities by accident. Instead, Microsoft architected those offline features in Outlook from
the ground up. Developers today need to think
about battery consumption. (Do you really need
to show that fancy animation when the battery
is down to 10%? Do you really want to start
data reorganization when the system has five
minutes of remaining power?) Developers also
need to understand network connections. (Why
does your application keep pinging the database
server when it is obvious that there is no connectivity and the application should use an offline
storage solution instead?) Developers need to
understand advantages and disadvantages provided by different devices. You shouldn’t assume
that the target machine has an 800x600 minimum resolution anymore, because with UMPCs,
the Microsoft Windows XP and Windows Vista™
operating systems run on lower resolutions. As
smaller devices become powerful enough to run
a Windows operating system (instead of a Windows Mobile® operating system), you’ll even see
more variation in screen resolution. Also gone
are the days when you optimize an application’s
UI for one type of input (such as either mouse
or keyboard input) only. Today you can use the
same interface in completely different ways, featuring anything from conventional keyboard in-
put, to pen interaction, to touch and writing with
a finger, to speech recognition.
This issue of CoDe Focus: Mobile PC Development is all about providing the best possible mobile PC experience for your users, and the most
competitive application for your business. It is
all about using the great new mobile PC features
found in Windows Vista. This new operating
system marks a big leap forward for mobile scenarios. Of course, mobility isn’t just about PCs
running Windows. You also need to consider
“the devices”. By that I mean special small devices running Windows Mobile, such as Pocket
PCs and Smartphones. This issue of CoDe Focus
isn’t about those. At least not until these devices
become powerful enough to run a full version of
Windows. But that doesn’t mean that Windows
Mobile development isn’t important. Quite the
contrary! In fact, CoDe Focus is considering a
special issue on that topic as well. Check out
www.code-magazine.com/focus for more information and let us know whether you are interested
in that particular topic.
Talking about sitting on your couch and reading a digital magazine: At EPS Software (CoDe
Magazine’s parent company) we have been hard
at work building a next generation publishing system based entirely on Microsoft .NET
Framework version 3.0, Windows Presentation
Foundation (WPF), and Windows Communication Foundation (WCF). By the time you read
this magazine, the first version should be available to the public, and this very magazine will
be one of the first to be published in this new
and exciting format, which aims to provide the
most advanced digital reading experience available anywhere. (And it is an open platform, so
ultimately other publishers will be able to use it
too.) We haven’t announced the final name of
the product yet, but since you are a technical
person who will probably really appreciate this
cool new technology, I’ll let you in on a secret:
visit www.code-magazine.com/ripple to be among
the first to get a copy of this free product, with
tons of content pre-loaded. Oh, and enjoy the
cool mobile PC and Ultra-Mobile PC features
we have built in…
Markus Egger
The slate.
The pen.
The mightier you.
When you’re ready to become mighty,
EPS Software can help you build
state-of-the-art Tablet PC and Mobile PC
applications. EPS Software employs
experts like Markus Egger (according
to Microsoft, one of the ten most important
Tablet PC influentials in the world),
Claudio Lassala (Microsoft MVP) and other
well-known names. EPS Software (we are
also the publishers of CoDe Magazine) has
what any organization needs to implement
a successful Tablet PC project.
www.eps-cs.com/TabletPC
online QUICK ID 0704022
Welcome from the Mobile
and Tailored Platforms Group
Welcome to our second CoDe Focus issue on mobile PC development.
T
his edition will focus on the mobile PC user,
their computing habits, and the ever-changing
demands they have of our applications. The
Mobile and Tailored Platforms team here at Microsoft® is focused on tailoring the Microsoft Windows
Vista™ Platform and experience through innovations in software and hardware. Our goal is to enable customer access to their digital information
anywhere, anytime, in new and exciting scenarios.
We, however, are only part of the solution as you
the developer will ultimately complete the overall
value proposition.
Frank Gocinski
[email protected]
Frank and his team focus on
ISV business development and
evangelism. They bring the
mobile message to the party and
help developer understand why
and what they should be doing in
order to make mobility pervasive
on the PC.
The WOW has Started
With the availability of Windows Vista in the marketplace, we see a mature Tablet PC platform that
is now a vital and central part of the Windows
operating system. In addition, sales of the broader
mobile PC category continue to grow and outpace
that of the desktop. Windows Vista brings a solid
platform for developers building mobile PC applications and a host of new features that make it
the “go anywhere” environment. The “Wow” starts
now.
Welcome from the Mobile and Tailored Platforms Group
The popularity of the new Ultra-Mobile PC (UMPC)
is helping to re-define the scenarios in which computing is possible and acceptable. With its compact
size, a touch-enabled screen, and the power of the
Windows Vista operating system, the UMPC becomes a perfect companion device complementing
your desktop (rather than replacing it) and bridging the technology gap between the phone and the
computer. The UMPC ensures all your information
and Windows applications are at your finger tips
whenever and wherever you want them.
Other new and innovative hardware is also hitting
the market now with devices that support the Windows SideShow™ Platform.
Windows SideShow
gives the user access to
information without
forcing them to boot up or start
up their computer.
Windows SideShow is an exciting new technology
that dramatically expands the usefulness of your
mobile PC or desktop computer. By closely integrating a variety of devices with your computer running Windows Vista, you can access information in
many new scenarios without opening up or powering up the PC.
Windows SideShow gives the user access to information without forcing them to boot up or start up
their computer.
This edition of CoDe Focus walks you through the
technology stack for building applications that target these new scenarios and devices. Your customers want simple, direct, powerful, and ubiquitous
access to information, and they can now get this in
your applications and on their terms.
Challenges Exist and Windows Vista
Responds
Let’s face it, today’s digital world is both exciting
and at times overwhelming. Because applications
and services for managing information have grown
in capability and richness, they require us to invest a
significant amount of time learning how to use them.
This is true on the desktop as well as the mobile environment. It’s true for developers as well as end users. Advancements in Windows Vista bring clarity
and greater simplicity to many of these tasks.
Mobile PC users face challenges every day that
desktop users do not usually confront. Desktop users typically have a reliable source of power and are
continuously connected to a high speed and reliable network. In contrast, mobile PC users must frequently manage transitions as they move from place
to place. Windows Vista addresses the unique needs
of mobile PC users by providing several new and
enhanced features.
Windows Vista offers a simplified and more reliable
power management experience compared to earlier
versions of Windows. Power settings help you easily
balance the tradeoff between power conservation
and system performance.
Resuming from standby or hibernate was not always consistently fast or reliable. Windows Vista
promotes the use of sleep states so you can use your
computer whenever you want, without having to
wait for the computer to start. As a developer, you
need to be aware of this as it will impact how your
application should handle this transition.
Customers use their mobile PC in more places and
in more ways than they can a desktop computer.
Windows Mobility Center puts the most frequently
used mobile PC settings in a single location. Now
you can change your display brightness, volume,
power plan, wireless network, external display (such
as a network projector or additional monitor), and
synchronization all from one place.
Presentation settings are options on your mobile
PC that you can apply when you’re giving a presentation. If you’ve ever had your display turn black
during a presentation, you will appreciate that your
screen saver automatically turns off every time you
give a presentation.
Microsoft Mobile PC Team
Anyone who has tried to connect a mobile PC to an
external display knows how challenging it can be.
Windows Vista simplifies the process by automatically detecting the external display, showing your
mobile PC’s desktop, and starting a wizard to guide
users through the use of an additional monitor.
Windows Vista also makes it easy to give a presentation from a mobile PC. By using the Connect to
a Network Projector Wizard, you can connect to
any Windows-compatible network projector over a
wireless or wired network.
The Tablet PC ushered in a new era of mobile computing: a single, fully functioning computer that is
practical and comfortable while you’re on the go
and also works well when you’re at your desk. With
integrated pen support, touch screen support, digital ink input, handwriting recognition technologies,
and innovative hardware, Tablet PCs are comfortable and productive in any place and at any time.
Now, Windows Vista includes significant improvements that make the Tablet PC easier to use and
that further enhance Tablet PC functionality beyond
that of notebook computers.
Visual feedback for single-tap,
double-tap, and right-click
(while subtle) gives you more
confidence about the operation
that you’ve completed.
Check boxes in Windows Explorer now make it easier to
select multiple files. When you
hover over any set of files with
a tablet pen, a small check box
appears. You can select one or
more files and then move, copy,
or delete them as a group.
Kurt Geisel
Product Unit Manager,
Mobile PC Platform
Windows Vista brings much
deeper integration of Tablet PC
capabilities into the operating
system. Now we are stepping
back to take a good look at what
we are doing for our developer
partners. We hope to make some
fast-paced investments, and we
look forward to seeing where we
can go together.
If you’ve ever had your
display turn black during a
presentation, you will appreciate
that your screen saver
automatically turns off every time
you give a presentation.
Pen Flicks are new gestures that you can make with
a tablet pen to quickly navigate and perform shortcuts improving the document and Web browsing
navigation experience.
Touch input is another great new feature, ripe for
your applications. Ultra-Mobile PCs benefit from a
touch-based UI as do the multitude of Tablet PCs
that now include this capability. Use your finger to
run Windows Vista; it’s simple, it’s intuitive.
Windows Vista also ships with an improved Tablet
PC Input Panel. You can use the writing pad or
the character pad to convert your handwriting to
text, or use the on-screen keyboard in the same
way that you use a standard keyboard. Entering
URLs, e-mail addresses, and other non-dictionary
words such as file names is faster and easier with
AutoComplete, a new addition to Tablet PC Input
Panel that works just like AutoComplete in other
programs.
Noticeable improvements have been made to the
handwriting recognition experience right out of
the box. Even so, Windows Vista includes two new
features that you can use to teach the handwriting
recognizer about your writing style and vocabulary.
These features can further improve handwriting
recognition accuracy. In Windows Vista, instead
of having to change your handwriting to get better handwriting recognition results, you can train
the handwriting recognizer to recognize how you
write characters and words. If
you turn on automatic learning
in Windows Vista, it enables
Entering URLs,
the handwriting recognizer to
e-mail addresses, and other
learn your handwriting style
or vocabulary by collecting
non-dictionary words such as file
data about the words you use
names is faster and easier with
and how you write them. The
personalization occurs behind
AutoComplete, a new addition to
the scenes, without user interTablet PC Input Panel
action.
that works just like AutoComplete
in other programs.
Windows Vista gives users one
place—the new Sync Center—to
manage data synchronization
between computers, between
computers and servers, and between computers
and devices. This capability has become increasingly important as the range of computers, devices,
locations, and data sources that customers want to
synchronize has exploded.
• Ms. Tara Prakriya—Architect on the MTP
team—has some visions of the future to share
with you in her “Into the Future” article.
• One of our Mobile PC MVPs, Frank LaVigne,
strokes some ink around our new InkAnalysis
API. Want to understand what a user’s written on that page? Parse it with InkAnalysis!
• Rod Paddock, who all of you faithful CoDe
Magazine readers know, spent some time on
the complexities of deploying your Tablet PC
applications.
• Our technical editor behind all the MTP developer content, Eliot Graff, debuts his writing prowess with a great “read this first” article on the development environment.
• And the first lady of Tablet PC development,
Ms. Julie Lerman, has done awesome work
on ASP.NET, AJAX, and Ink.
All in all, I think you’ll enjoy this issue and won’t
be able to put it down until you read it word for
word.
Let us know what you think. Visit our Developer
Center on MSDN (http://msdn.microsoft.com/
mobilepc) and send your comments to us at [email protected]
Remember—Do it on the road and do it in ink!
Frank Gocinski
Writers in this Issue
Inside this Focus edition of CoDe Magazine, we
have an awesome selection of rich technical articles for you to learn about the platform services
available to push your applications into the mobile
PC space.
• Our colleagues at Intel have written a great
article on developing games for the UMPC,
we hope you enjoy it and build some cool
games.
• Todd Landstad—our MTP ISV Architect—put
together the bible on development for the
UMPC platform.
• Dr. Neil Roodyn graces these pages with his
insights around adopting your existing applications to UMPC and a second article on
Windows SideShow development.
• Hilton Locke—Tablet blogger extraordinaire—
has put his fingers to work not testing touch, a
difficult task for this tester, but writing about
touch development.
• Nick Landry’s article, “Mobility in Windows
Vista” is a definite must-read.
• The .NET Framework legend, Billy Hollis,
shares his words of wisdom around WPF and
Ink—very cool.
10
How to Be Where
Your Customer Wants to Be
We all know that applications have evolved, and not just towards
Web deployment, .NET Framework development, and mash-up
functionality. Applications are also evolving in terms of where and when customers
are accessing their applications, their data, facts, and other computing resources.
Tara Prakriya
[email protected]
Tara Prakriya is an Architect for
the Microsoft Mobile & Tailored
Platforms Division. Tara started
at Microsoft working on the MSN
Platform in 1997. During her
tenure there she oversaw early
adoption of the .NET Framework
as the Product Unit Manager of the
content management platform.
She was also a Program Manager
on the ad system and the content
and user data warehouse. Joining
Windows in 2002, she was the
Product Unit Manager for Mobile
PCs in Vista and then the Product
Unit Manger for Tablet PC in
Vista. Most recently she has been
incubating projects in emerging
markets that leverage pen, touch,
and ink. Prior to joining Microsoft
she worked at Merck & Co’s US
Human Health Division IT focusing
on financial data warehousing.
F
actoring in the context of where users are and
what they’re doing when they use your code
will ensure your application is meeting all your
customers’ needs. Because of its versatility, when
you develop an application to deploy on a Microsoft® Windows® computer you should consider aspects such as:
• What additional tasks users are doing at the
same time.
• The dimensions of the PC they’re interacting
with.
• The ambient distractions in the environment.
• The amount of time and concentration available for the tasks at hand.
Chances are you architect your application that
typically has an on-line component (such as one
needing services from the Web) so that it performs
when isolated from that source of data. That’s good,
because to build good on-line applications you need
to build awesome off-line applications. Not only
can connectivity be intermittent and interrupted, so
are the user’s usage patterns, which have evolved
from long periods of work to bursts of usage. And
do you really want to write the application that runs
the user’s battery from 8% down to nothing during a major synch, because you forgot to cache data
in case of low battery? Sure, there’s a lot to think
about, but these are also opportunities to shine.
Variety of Locations and Interactions
Windows-based computers are no longer just beige
boxes sitting under desks. Cutting-edge hardware
like the Toshiba Portege R400, Motion LE1600TS,
Lenovo Thinkpad X60, HP TouchSmart IQ770 PC
(Figure 1), and TabletKiosk eo (Figure 2) signal a
Frank Gocinski
[email protected]
Frank and his team focus on
ISV business development and
evangelism. They bring the mobile
message to the party and help
developer understand why and what
they should be doing in order to
make mobility pervasive on the PC.
12
How to Be Where Your Customer Wants to Be
Customers—and we,
ourselves—now demand and
expect instant access to
everything on a computer without
having to drop everything and
focus to interact.
new age of computing where users will find a Windows-based device in kiosks, kitchens, and even
coat pockets. Developers should consider two implications of these additional locations:
• Ease the user experience to address technology challenges inherent in mobile computing.
• Create new innovative applications and services to cater to needs that come up in the context of a customer’s life away from the desk.
Technology challenges include the need to bridge
connectivity outages, multiple power states, and
direct interaction with the screen, as in the case
of Ultra-Mobile PCs. Creating new applications
means responding to the customer’s opportunities through quick bursts of interactions to get upto-date, glanceable reference information, and to
make quick updates or pass time casually. Applications which are thoughtful of these new technological challenges of mobility and respectful of the
new usage patterns of the customer are the new
best-of-breed of tailored applications.
As you read through this edition of CoDe Focus,
you’ll see many scenarios, many technological approaches, many development hints, and a great
overview of how the Windows Vista™ operating
system helps you and your customers in this truly
mobile, truly tailored world of computing.
And it’s a good thing. Customers—and we, ourselves—now demand and expect instant access to
everything on a computer without having to drop
everything and focus to interact. By taking advantage of the new platform and out-of-the-box features in Windows Vista, such as how the Search box
appears in folders, you can deliver what your customers expect—and more. When you can anticipate
your customers’ needs and tailor your applications
to provide ease of use in specific scenarios, those
customers will view your applications as more than
good—they will see them as indispensable.
Tailoring Applications
Central to an understanding of the technologies provided by the Windows Vista Platform is a sound understanding of the features that we expect to see in these
tailored applications; a few to consider are listed here.
They very well may need to come back to ideas,
information, and status at a later time. This means
updating application state while being mindful of
disk access to save battery and expecting network
connections to be frequently dropped.
Quick Data and Note Capture for
Later Analysis and Retrieval
Figure 1: The HP TouchSmart IQ770 PC.
Collaboration
Collaboration is about sharing information with the
person next to you or a teammate on another continent. With a Tablet PC, sharing may mean turning or angling the screen to share baby photos. And
while going through the photos, capturing quick ink
annotations right on the screen to indicate who is
in the photo and add interesting captions. As a developer, you might write the application as a Web
service where the photos are available locally in
cached mode. The user can upload the ink annotations whenever a strong network connection becomes available. Because the Windows Vista Platform can recognize ink in the background, the user
can elect to save recognized text in the background
and then add the results to the JPEG metadata for
easy searching. If the computer has a low battery,
your application can default to display the photos in
lower resolution with a thicker black border.
Multitasking with the Environment
It’s common to see customers physically and mentally juggling various items they have with them:
phone, keys, briefcases, book bags, to-do lists, complex project plans, and the like. It’s the same when
customers use a computer. A well-developed application can take advantage of gestures to quickly implement commands. For example, while accessing
maps on a UMPC in the car, an application should
make it easy to accept ink written anywhere on a
screen and simple gestures to process the location
desired. Or the developer can create UI that fits
the situation, like big buttons for easy touch finger
targeting and straightforward defaults that demand
little interaction from the customer. Also, ensure
the application and service is easy to see and use on
smaller screens with lower resolution.
Time-Slicing Activities
With all that juggling, don’t expect customers to
complete a task fully before starting the next one.
Add unstructured note taking and file markup
functionality to your scenario by taking advantage
of out-of-the-box Windows Vista
functionality for any mobile PC
with a digitizer. Windows Journal is
… don’t expect customers to
as flexible as a piece of paper with
the added advantages of searchable
complete a task fully
ink and easy adding of “important”
before starting the next one.
flags. Users can read documents
and customers can annotate with
personal comments if the customer
prints to Windows Journal through a printer driver—which means organizing projects with supporting paperwork electronically in one place is actually
feasible.
Core Customer Scenarios
We have done a lot of research with our customers to try and define some core areas and scenarios
for which they want and need to use their computers more times of the day. Taken together, we often
hear: “Give me more applications to help me when
I am traveling, shopping, on the train or bus—basically, fill the gaps where I know access to information can help me.” Consider these exciting areas
when thinking about projects.
Personal Productivity
Customers want to access their e-mail anywhere
and at any time. They want easy-to-use software
that lets them build a shopping list while they’re
waiting in line at the bank. They want to see where
the traffic bottlenecks are before they get on the
Figure 2:
The TabletKiosk
UMPC.
13
road, and they want the information to plan an alternate route while they’re stopped in traffic. Their
need to access personal productivity tools shouldn’t
end when they leave their desk. Your applications
should assume that your users are mobile.
Your applications
should assume that your users
are mobile.
Thru Car Navigation
Users want to say “I don’t need to stop for directions” with authority. They want to know that they
can find and get to the marine sanctuary, if that’s
what their spouse wants to see that day on vacation.
Or perhaps they want to find the best place nearby
for a celebratory milkshake while they’re watching
their kids play soccer.
Social Communities
As social communities become more and more a
part of life, people will want pervasive access to
their contacts. What good is an instant message if
you receive it three days later? And with increased
location awareness (such as GPS), a user’s computer should tell them when they are near a friend or
contact who have signed up to let them have that
information.
Home Automation
Perhaps users want to control their home automation settings from a UMPC while on the couch or
as they get ready for bed. Or maybe they want to
change the temperature at home if they’ve already
left on a trip and they realize they forgot to turn
down the thermostat.
Games
Imagine users being able to continue play on your
latest game while they sit on the bus or at the airport. Designing games that allow users to play them
in short bursts requires a different approach than
designing them for extended multiplayer interaction.
Media and Entertainment
Want to watch the last episode of your favorite reality show that you saved to Windows Media Center? Do you want pervasive access to your satellite
radio, from home to the car to work? Maybe you
need navigation information to a movie or concert?
Imagine extending your home media system to automatically integrate with your car’s systems. Your
customers want this too, and they are willing to pay
for the ability. The features available to developers now make building these sorts of applications
within reach.
Seize What’s Before You
We’re embarking on an amazing odyssey through
time and space; the scenery is changing, the customer is demanding, the opportunities are endless.
We’re excited about what the Windows Vista Platform has to offer to enable the tailored application
to take hold. We’re going to be watching as developers and ISVs work with us to change the world.
Don’t underestimate the opportunity to provide applications and services for customers who have just
discovered they’ve got time to kill due to a delayed
flight, a longer than expected wait at the doctor’s
office, or while seated at dinner waiting for ...YOU!
to seize the opportunity.
We’d like to extend a personal note of thanks to
a few people who contributed to this article: Eliot
Graff, Lora Heiny, Todd Landstad, and Tina Snyder.
Tara Prakriya
Frank Gocinski
Education
Applications that promote mobile educational tools
allowing students to travel from the classroom to
the library to home presents an exciting area to help
change the world! As calculators did in the past, the
mobile PC can become a tool to help students at all
grade levels do better in school. Applications that
help teach handwriting, character-based languages, drawing, painting, surgery—these are just a few
possibilities when a school system employs stylusbased computers. Distance education, accessibility,
and collaboration are just a few of the pedagogical
hurdles that mobile PCs can help conquer. You can
imagine endless scenarios for mobile PCs in education, and the technology is now available.
14
Developing Windows Vista
Applications for the
Ultra-Mobile PC (UMPC)
Growing up in the 80s, two of the concepts that drew me in to
computer science were computers that were small enough to
take with you (called “Luggables” then) and displays that were
thin enough to hang on a wall. Technology has come a long way in that
time—and the Ultra-Mobile class of personal computers that run the Windows Vista™
operating system is a great example of that progress. So what is a UMPC and why
would I be interested in writing an application that is optimized for it?
M
icrosoft defines the Ultra-Mobile PC as an
ments over the course of the next few months, as
ultra-small form factor Windows Vista PC
well as the addition of built-in keyboards included
that leverages mobile
in many of the new designs.
features for its design. UMPCs
Fast Facts
weigh less than two pounds,
feature a 7-inch or smaller
What? No Keyboard?
Ultra-Mobile PCs run Windows
touch-enabled display, and
Vista and are a great choice
have connectivity features
Many of the current UMPC
when your users are on the go.
available (Wi-Fi b/g, Bluedesigns are slate models, which
The
Ultra-Mobile
PC
has
a
few
tooth®, USB 2.0, and GPRS),
rely on the touch screen for inunique features that make it
so these machines can bring
put, but several models include
the perfect fit for current and
your products to places you
a built-in keyboard and a mouse
new Windows
never considered before.
thumbstick for more traditional
input. In either case, the input
Vista mobile scenarios.
When people first see a UMPC,
methods on the UMPC provide
they often ask, “What operata natural way to interact with
ing system is it running?”, which is pretty natural
the mobile scenarios mentioned above. The UMPC
due to the UMPC’s small size. Whenever I talk
uses one of the following input methods:
about UMPCs I am talking about the full Microsoft® Windows® operating system. Will these be the
fastest Windows Vista machines you can buy? No,
Touch Screen (Select with a Finger)
at least not in the short term. However, they are
fully capable of running the desktop Windows Vista
From the user as well as an application’s point of
applications that you rely on and provide a great
view, the touch screen works just like the mouse.
mobile experience.
The digitizer interprets tap and double-tap on the
All UMPCs meet these specifications:
screen as mouse clicks and double-clicks. Movement of the cursor is also the same, but since you
• Intel® or Via™ CPU running at least 900 MHz
use your finger or a stylus to tap you no longer need
• 7” widescreen display or smaller
to move the pointer from one location to another—
• Integrated video card with up to 128 MB
you just move your finger.
RAM
• 2.5” HDD with up to 120 GB of storage
• Up to 1 GB of SDRAM
Stylus (Type or Write with a Pen)
• USB 2.0 ports
• Optional ports for video out and media cards
Windows Vista has Tablet PC technology built in,
• Integrated GPS and/or camera
so the UMPC can leverage the operating system
• Integrated high-speed cellular data (such as
components such as the handwriting recognizers
GPRS)
and Tablet PC Input Panel previously found only
on Tablet PCs.
This is, of course, the current specifications at the
time of this article and varies between manufacturFor the end user this means they can use the styers—you’ll see speed, power, and feature improvelus to handwrite information right on the screen.
Todd Landstad
[email protected]
Todd Landstad is on the ISV
Business Development Team for
the Microsoft Mobile & Tailored
Platforms Division. He works
with Windows Vista ISVs to help
design and ship great mobile PC
solutions.
A Canadian-born transplant to the
great Northwest, he enjoys time
with his family, woodworking,
and tech gadgets.
Developing Windows Vista Applications for the Ultra-Mobile PC (UMPC)
15
The engine converts that data to text, whether it is
a Web site address or an e-mail message, and sends
it to the application.
The fact that Windows Vista contains Tablet PC
technology also opens up the ability for applications to offer ink-based features such as ink gestures, flicks, and handwriting areas in applications
without having to take on dependencies—it’s all in
the operating system.
Hardware Controls (Navigate and
Select with Buttons on the UMPC)
Mitica Manu, PhD
Test Manager,
Tablet PC Team
What I really love about my
job is the unique blend of a
startup environment (yes, in a
huge company!) and bleeding
edge technologies. In college, I
learned about neural networks
as something exotic; but here
at Microsoft, I really put them to
work. I never stop learning new
things from the unique and great
people that I work with everyday.
Every UMPC design includes a set of hardware buttons to complement the touch screen. These buttons vary from model to model but include some
combination of a directional pad, enter key, page
up/down, mouse left/right buttons, and a set of
user-configurable buttons.
Users can configure some of the buttons so they can
choose what function they perform. For example, a
user might reconfigure buttons to play a game that
allows the user to select which buttons allow a character to move, attack, or interact in a role-playing
game.
Optional: Add on Bluetooth
or USB-based Keyboard
Because the UMPC design includes USB 2.0 ports
and Bluetooth connectivity, you can use any compatible accessory including keyboards and mice.
Devices like Bluetooth headsets and speakers, GPS
devices, and cameras are also
great pairings.
UMPCs weigh in at less
than two pounds with a 7-inch or
smaller touch-enabled display and
have all the connectivity features
available (Wi-Fi b/g, Bluetooth,
USB 2.0, and GPRS).
What Can I Do with a
UMPC (That My Laptop
Can’t)?
In the spare time I have away
from work, I love woodworking. I learned a wood shop lesson long ago—always have the
right tool for the task. This rings
true for PCs. While it may seem
that for mobile computing the laptop fits the bill every time, when you really look at it there are many
mobile scenarios that go beyond the laptop. In fact,
any scenario beyond the conference room is a case
in point.
When you boil it down, the Ultra-Mobile PC has a
few unique features that make it the perfect fit for
current and new mobile scenarios:
1. Very small size. If you travel much, you’ll
probably agree that in comparison to travel-
16
ing with a full-size laptop, taking a sub-twopound device with you is a dream. While the
current UMPCs typically weigh between 1.7
- 2 pounds and feature a 7-inch, widescreen
display, manufacturers will eventually offer
models that are both lighter and smaller (4.8inch display).
2. Integrated touch display. Consider a scenario
where you try to use a mouse or a notebook
touch pad while standing on a subway. In
comparison, a touch screen on a UMPC makes
your commute more productive. The first time
people use a touch digitizer they might be little
intimidated. They usually ask where the keyboard is. Once they realize that touching the
screen acts just like the mouse, they find it intuitive to use a UMPC. It’s true that you won’t
want to write a long article, develop applications, or compute pi to the nth decimal point
on a touch display, but it’s all about choosing
the right tool for the task.
3. Low cost. With any new platform, you’ll see
higher costs at the early stages, but as hardware manufactures launch additional models
you should see significant drops in prices of
UMPCs. In part, UMPCs use standard, offthe-shelf components (display, RAM, HDD)
to help to reduce the manufacturing cost and
make UMPCs a great companion device.
4. Long battery life. The current standard battery configuration runs Windows Vista for 2-3
hours but many manufacturers offer extended
batteries to last you more than 5 hours; ample
enough time for a decent plane ride or work
day with an appropriate power plan. There
will always be an opposition between size and
weight against longer battery life. The trick is
to make use of standby/hibernate and use applications that are optimized for the mobile
experience.
The combination of these unique factors with the
fact that this platform is running the exact same version of Windows Vista as a desktop or laptop means
that the UMPC is another great tool to use for accomplishing mobile computing tasks.
Mobile Scenarios
Let me walk you through some of the mobile scenarios that make the Ultra-Mobile form factor the
right tool for the job. Many of these scenarios focus
on consumption of content with less of a focus on
creation, which is intentional. This isn’t to say that
you can’t create documents, rip music, or edit videos on the UMPC, but much of the value in truly
mobile scenarios that I am covering is related to
consumption of data.
Media and Entertainment
With its clear, bright display, I can have a great
movie or TV experience on the UMPC. The ability
to record television shows on the device via Win-
dows Media Center, for example, and then take
those shows with me really salvages a boring train
commute. With all the downloadable video content
on the Internet these days, having a UMPC makes it
easy to purchase movies for download as well.
With the integrated WWLAN capabilities, Wi-Fi,
or Bluetooth via a cell-phone data connection, the
UMPC makes streaming media possible on the go
rather than just at your desktop. As you’d expect,
the quality for streaming video may limit the bandwidth of the GPRS data connection, but audio is
a perfect match, making the UMPC a great way to
listen to your favorite Internet radio station wherever you are.
Digital Memories
Digital photographs are such a part of our everyday
lives that you often don’t do anything more with
them than send, view, and print them, right? A
UMPC makes it easy to accomplish even more. For
example, I take a lot of digital photos and now have
so many that I need a way to search through the
mass to find that one good vacation photo. Adding
keywords to photos has been around for a while,
but Windows Vista and its photo gallery makes it
easy to add them—now called Tags—and search for
the photos by tag when I need to find one.
The features of the UMPC make it a good match for
adding value to the digital photo experience due to
the good sized display, small form factor, and touch
screen. I regularly use my UMPC for these digital
photo-related tasks:
• Photo display: Either on my desk at work or
the coffee table at home, if I’m not using the
device, it’s showing full-screen photos of my
kids.
• Handwritten photo keywords: You already
know how to add keywords, but what about
the ability to handwrite on the photo to add a
keyword or star rating? You don’t need to alter
the original photo; just recognize the ink, add
the strings as keywords to the image data, and
you have a quick way to tag photos on the go.
• Searching tagged images without a keyboard:
An extension of the ink-based tagging enables
an ink search feature. Ink-based searching allows you to use ink to create a keyword search
for images with the handwritten word. The application could even present suggestions for
the search term while you’re writing it. Writing
the letter “M’ for example could produce all
tags beginning with that letter, and then narrow the list as you write additional letters.
Games
Computer games are one of the driving factors
in hardware innovation and performance. At any
given time, many games require the latest graphics cards and CPUs to keep up with the realistic
textures, shading, and lighting in the virtual worlds
they so amazingly present. I don’t expect this to
change in the near future since many of the video
cards in these machines are the size of the entire
UMPC itself. This doesn’t eliminate all the other
games running on Windows however.
Generally, the limiting factors for running a highend game on a UMPC will first be gated by videocard capabilities such as Microsoft DirectX® hardware and software rendering. Next is any game-specific resolution requirements followed by security
restrictions like needing the DVD to be inserted
during game play. High-end hardware requirements
do not make for a great mobile scenario.
However, I love playing games on this platform.
I’m not big on the high-detail, super-realistic games
with dozens of keyboard commands that take hours
to figure out. I like the games that offer great game
play and rich colors and sound that entertain me
when I have a few minutes here and there. I call
these casual games. They make great use of screen
real estate and I can pause them if I’m interrupted.
I also look forward to games that take advantage of
the mobility of an Internet-connected device that
I can easily take with me and that is running Windows Vista. Network-aware, multiplayer games that
even take into account my location (via integrated
GPS or IP address) and that allow me to interact
with touch and on-screen gestures could add a
whole new genre to Windows gaming.
Dustin Hubbard
Product Unit Manager,
Ultra-Mobile PC
For me, creating exciting UMPC
software, such as the Origami
Experience, and connecting
to customers through our
community site—origamiproject.
com—is extremely rewarding.
Imagine a driving game that knows the city you
are currently in and sets the race course to be in
that matching location, a trivia game that orients
its content to points of interest in your location, or
a puzzle game that simply chooses images for its
background to match your locale are all interesting
possibilities.
Navigation / Location-based
Scenarios
Speaking of location awareness—there’s an entire industry built around this scenario. Dedicated
GPS and navigation devices are great, but what
about getting real-time traffic, point of interest, or
restaurant information right off the Internet? The
combination of the UMPC’s features (including the
integrated GPS) and connectivity provides this set
of information.
A simple Web query can pull any
kind of contextual-based piece
of information imaginable off
the Internet to help navigate this
busy and always changing world.
Combining that with Wi-Fi or
GPS location and access to the
user’s calendar can provide great
context for narrowing a large
amount of information.
With the integrated WWLAN
capabilities, Wi-Fi,
or Bluetooth via a cell-phone data
connection, streaming media
is now possible on the go rather
than just at your desktop.
17
Younger students do have the capability of using a
pen, but have not learned how to use a keyboard.
An increased speed of interaction can occur with
the ability to write or draw answers to questions.
For math applications, scenarios range from the
user’s ability to point at and count items to create
handwritten formulas or manipulate data in a graph
using their finger.
With current ink-enabled, classroom-based learning
applications, teachers could monitor their pupils’
progress and highlight answers to share with the
class.
The natural input with the pen allows the teacher to
see more than the answer from the student. The teacher can see handwritten formulas, diagrams, or even
scratched-out ideas that may show the thought pattern
of the student. In addition, normally quiet students
could have their answers highlighted when they may
have been otherwise overlooked without this model.
Figure 1: Origami Experience opening screen.
Education
From grades K-12 through college, the UMPC not just
adds some value but improves on current scenarios.
The form factor, touch digitizer, and low cost lead
to features that work well in educational environments. Children in lower grades can obviously appreciate the small size of the device, being able to
handle it easily; but the natural interaction of the
touch display is what improves on traditional software applications.
The form factor,
touch digitizer, and low cost
lead to features that work well in
educational environments.
I can’t imagine a better way to
learn to write than instant feedback on the shapes of written
letters. Some applications will
play a sound or show the object matching for the word that
was written (for example, the
application might show an image of an apple for the correctly
written word).
In higher-education levels, students could download curriculum over the Wi-Fi connection, take
notes about classes, and mark up presentations with
handwritten annotations. The ability to “grab on”
to data on the screen using touch and manipulate it
could help students understand the cause and effect
of changing the data.
Windows Vista Applications that Are
Optimized for Mobile Scenarios
In many cases where the talk is about Windows
Vista applications that are optimized for touch or
small displays, often the talk turns to “lifestyle applications.” Whatever we call these applications, I
think of them as applications that perform a nonbusiness task or set of related tasks which add value
for users when they are away from their desks. The
obvious key point here is being mobile.
If you search the Internet for “lifestyle applications”
you’ll find that there aren’t very many of them.
What you do find are health, weather, or travel-related applications—showing how these types of applications are just starting to take shape.
As I mentioned earlier when talking about the mobile scenario strengths that start to appear while
using the UMPC platform, Media and Entertainment is a category that immediately shows value. A
UMPC has a big hard drive to hold my music and
videos and I usually have a decent amount of time
during my commute or on a plane.
Figure 2: Origami Experience programs.
18
There’s no reason why I can’t run the standard
desktop media software on my UMPC. However,
there are some drawbacks to the high-density data
and small target areas of a traditional media library
or tiny transport controls (play/pause buttons)
when walking around using the device or traveling
anywhere beyond my stationary desk.
There are a few applications out there that show
how slight optimizations in their UI or feature design can make a real positive impact on the mobile
solutions they are presenting.
I want to highlight some of the features in current
applications so that I can show which areas to pay
attention to when thinking about making optimizations in your own applications.
Origami Experience
The Origami Experience for Windows Vista (http://
www.microsoft.com/windows/products/winfamily/umpc/default.mspx) is a great example of an
application optimized for the UMPC platform. It
runs full screen, allows access to your applications
and media with a touch-friendly UI (no double taps
or context menus) and displays system information
that you can read in a glance.
Figure 3: Origami Experience media selection.
One of the features I like most in this application is
that you can get to anything you want in a click or
two. Not only that, but you have system information
and currently playing media always visible.
The Origami Experience also combines related topics into one application so a user can listen to music
while watching a slide show, for example. Let me
highlight a few of its features:
Home Area
Figure 1 shows the Home area of the Origami Experience. Right away, you can get a clear idea of what
functionality it provides. The Origami Experience
includes Music, Videos, Pictures, and Programs as
the main set of buttons on the screen. It also clearly
presents Settings and Help.
In almost every screen within Origami Experience,
you have immediate access to the playback controls
at the bottom and the title bar controls at the top.
In addition to the typical media controls, you’ll also
see that the last song or video that you were playing
is already queued up (whether an individual track
or playlist), so you can resume playback immediately after launching the application.
Along the top of the screen, the Home button always
returns you to the Home screen from wherever you
are. The top of the Home screen also contains the
Window Switcher button, battery status, network
status, and current time. Each button performs the
associated task or opens up the appropriate control
panel area when tapped.
Figure 4: Origami Experience picture settings.
allows you to create multiple categories with shortcuts to applications, files, folders, or Web sites.
Music, Pictures, and Videos
When you open Music, Videos, or Pictures (Figure 3),
you’re taken to the Now Playing screen. This screen
shows you what’s currently queued up for that media
type (based on what you previously played). There’s
also a scrollable list of all of your playlists (Music),
all videos (Videos), and slide shows (Pictures). Tapping one of the items in that list
immediately starts playing the
selected item.
Programs
Settings
Similar to the Program Launcher from the Touch
Pack (the UMPC navigation UI that was released
through OEMs on UMPCs running Windows XP
Tablet PC Edition), the Programs view (Figure 2)
The Origami Experience continues its single-tap user interface into its settings screen
(Figure 4). A user can config-
The Origami Experience
for Windows Vista is a great
example of an application
optimized for the UMPC platform.
19
ure the appearance, program shortcuts, and media
libraries from this area.
UMPC Design Guidelines
In order to present the suggested means to creating an optimized UMPC application, Microsoft has
published a set of general guidelines on the Web
for Mobile PC applications that has a section specifically targeted at the UMPC platform. You’ll find
these guidelines available in the Mobile PC section
of the Win32 and COM section of MSDN. You can
also find them through the Mobile PC Developer
Center, http://msdn.microsoft.com/mobilepc. To save
you from having to search the Web, I’ve included
the seven major areas in the guidelines below:
Designing a Great Ultra-Mobile PC
User Experience
At minimum, your Ultra-Mobile PC application
should do the following:
• Be easy to use and fully functional in the absence of a standard input device, such as a
mouse and keyboard.
• Present information in such a way that users
can digest it at a glance. Avoid an overly dense
user interface.
• Launch and run in full-screen mode.
• Provide exit control while in full-screen mode.
Principle
Description
Simple and direct
Don’t try to do everything—do a few things very well.
Enjoyable and
desirable
Invest in visual appeal and excitement.
Personal
Meet individual user's needs by designing an application
in which altering the configuration is easy. Consider
interfaces that are adaptive to different peripheral
accessories and locations.
Glanceable
Create a user interface that does not need to be studied.
Users should be able to read the display at a glance to
quickly gather necessary information in situations where
their attention isn't fully focused.
Efficient
Enable quick, convenient access to information and tasks.
Minimize the number of displayed options and settings.
Casual
Optimize for users to consume rather than create content.
Ensure quick accessibility and avoid overly dense displays
of information.
Consistent
Maintain consistency with the Windows Vista User Experience
Guidelines (http://msdn.microsoft.com/windowsvist/
uxguide).
Extensible
Take advantage of the Windows Vista development
environment, UMPC-specific tools, and UMPC-specific
guidance to develop a rich software experience.
Secure
Ensure that users feel confident that personal data is safe
and private.
Table 1: Design principles for an optimal Ultra-Mobile PC user experience.
20
• Run in both portrait and landscape modes.
• Ensure that all user interface elements are visible and unclipped at 800 x 480.
• Ensure that dialog box height is less than 480
pixels.
• Be designed for multiple resolutions and display sizes. Ensure that user interface elements
scale to accommodate a range of display sizes
and typical display resolutions.
• Disable or eliminate all user interface elements
that are not required for a specific task, including captions and status bars.
User Experience Design Principles
The Ultra-Mobile PC is particularly well-suited for
applications that emphasize content consumption
over content creation and that give users spontaneous and precise command and control. Quick,
simple, and direct access to rich and engaging content is the hallmark of a great Ultra-Mobile PC user
experience.
Design Principles
Use the set of high-level design principles in Table 1
to address user scenarios and guide feature development for Ultra-Mobile PC software.
Additional Recommendations
• Use a mode that incorporates full-screen displays and asks users to accomplish one task at
a time.
• Invest extra time to create a well-designed,
complete user interface.
• Design applications that encourage personalization.
• Investigate opportunities for extending the user
experience through specialized hardware.
• Optimize the application for touch interaction.
Optimizing Touch Command
and Control
A well-designed Ultra-Mobile PC user interface
provides users simple, direct, and reliable touch interaction. Design and organize touch targets, such
as buttons and links, in such a way as to maximize
legibility, accessibility, and precision.
Touch Target Recommendations
• Make touch targets at least 1 cm x 1 cm.
• Scale targets to accommodate a range of display sizes and resolutions.
• Space targets sufficiently to ensure an accurate
interaction layer.
• Ensure that your target's graphics maintain
legibility. Keep in mind that graphics don't
have to fill an entire target area in order to
maintain legibility.
• Ensure that targets are not obscured by the finger or hand when the user is interacting with
the display.
• Provide larger targets when your application
will be used in limited attention scenarios,
such as driving or walking.
• Create an aesthetically organized touch experience by grouping targets by related functionality.
• Avoid placing targets at the edges of the display, where touch targeting can be difficult.
Targets that are positioned at the edge of the
display should have additional buffering so
they can't be moved off the display.
• Make your authentication requests predictable. Avoid prompting users during tasks or at
other times when they do not expect a security
prompt.
Designing for Mouse-free Scenarios
Like standard keyboards, mice are optional UltraMobile PC command and control devices. A good
UMPC user interface assumes that users are likely
to rely on alternative input devices, such as touch
and stylus.
Hover Feedback
Designing for Keyboard-free
Scenarios
While some Ultra-Mobile PCs have built-in keyboards, and all are keyboard-extensible, you
shouldn't assume that users will always have access
to a conventional keyboard. Even when a keyboard
is available, using it can inhibit the kind of spontaneous, flexible command and control users expect
from their UMPCs.
Security authentication is a further consideration in
a keyboard-free environment. In mobile situations,
it can be challenging for users to manage authentication dialog boxes with or without a keyboard.
Password entry is a process that requires the user’s
full attention to complete successfully. Password security may be compromised by visual feedback from
soft key events. For these reasons, you should think
about how, when, and where your application requires security authentication.
Keyboard-free Recommendations for Text Input
• Provide on-screen alternatives to commonly
used keyboard shortcuts, such as Copy, Paste,
Undo, Save, and Open.
• Provide on-screen alternatives to keyboard
modifier keys, such as CTRL, ALT, and SHIFT.
• Where text is required, offer user assistance,
such as autocomplete from history, MRUs, or
predefined “quick text.”
• Investigate ways in which alternate control
functionality, such as speech recognition,
might replace keyboard input as well as expand your application's capabilities.
• Consider providing specialized features, such
as GPS navigation and video conferencing,
through peripheral hardware accessories and
OEM-installed hardware.
• Use hardware control buttons to simplify user
navigation tasks; however, hardware buttons
may be specific to a particular OEM-configuration.
Security and Login
• Use authentication dialog boxes as sparingly as
your application's security requirements allow.
In the absence of a mouse, the Ultra-Mobile PC
touch screen does not provide hover-like feedback.
As a result, ToolTips and other information that
typically display as a result of a mouse-over hover
state are not accessible on the display.
• Don't rely on hover feedback for critical
tasks.
• Clearly label all user interface elements, or use
icon graphics to communicate meaning.
• Consider providing hover feedback with object focus after an item is tapped.
• If you use pop-up dialog boxes to provide hover feedback, make sure to provide a way for
the user to quickly close them.
Contextual Menus
Without a right-click mouse or stylus event, it isn't
possible to open standard contextual menus.
• Minimize overall reliance on right-clicking.
• Don't require that an action be accessible only
by right-clicking.
• Consider opening context menus with object
focus after a user taps an item.
Scrolling
Scrollbars require fine motor control and focused
attention to target and manipulate. Users may find
them difficult to manipulate by touch.
• Make scrollbar thumb and up/down buttons
at least 1 cm x 1 cm.
• Consider using paged or draggable content as
alternatives to scrolling.
Drag-and-Drop
• Finger dragging, gestures, and direct manipulation of content and user interface elements
should be quick and simple.
• Ensure that dragging by using a finger or gestures is discoverable.
• Include drag-and-drop functionality only
when the drag source and drop target are both
on the screen.
• Ensure that the user’s hand will not obscure
critical user interface elements while dragging.
21
• Ensure that a user does not need to have fine
motor control to initiate a drag.
Optimizing the Display Area
Ultra-Mobile PC applications should present information in a form that users can easily digest at
a glance. Overly dense content is difficult to read
quickly in mobile situations, and a dense user interface layer requires a great degree of user attention.
Display Recommendations:
• Design with information density in mind. The
less dense your presentation is, the easier it is
to read.
• Provide a means to hide or
eliminate common desktop
window controls, such as
This is a great time
resize and restore controls,
multiple toolbars, and multo make one point clear:
tiple panes.
you don’t need a UMPC device
• Consider scaling. Can the
user interface be hidden at
in order to develop applications
the lowest resolution?
to run on one.
• Make user interface elements
large enough to quickly understand and interact with.
• Minimize reliance on small targets, such as
spillers and spinners.
• Minimize reliance on menus and lists.
• Launch your application into full-screen mode
by default.
• Dedicate the full screen to one task at a time.
• Avoid complex tasks where possible. Break
up complex tasks into a sequence of simpler
tasks.
• Design so that a single tap on the screen is sufficient. In scenarios in which a single tap isn't
feasible, provide a light, user-interface layer or
overlay that will give users access to the full
control layer.
• For applications that do not display user interface elements in full-screen mode, such as
picture displays, games, and videos, a tap on
the screen should display controls to close or
reveal the caption bar.
• Make media and other rich content applications open, with minimal or no user interface
controls displayed. For such applications, ensure that there is a way for the user to access
controls that are necessary to exit the application and to change configuration settings.
Using Notifications Well
Status information is particularly critical to the Ultra-Mobile PC user, because the UMPC is likely to
encounter frequent changes in environmental and
system conditions. However, notifications require a
greater shift in user attention on the smaller UMPC
display than they do on larger desktop or laptop
screens. Balance status information requirements
22
against the risk of distracting users with invasive
notifications.
Notification Recommendations
• Use invasive notifications for critical information only.
• Provide hyperlinks in notifications so users
can navigate directly to related tasks or information.
• Highlight ambient status information.
Developing Applications Targeted
for the UMPC Platform
In developing applications for the UMPC, you
are potentially just developing robust applications for Windows Vista that are optimized for
mobility, touch, and varying display sizes. Your
efforts here are not limited to just the UMPC
platform.
I am applying many of the Windows APIs to writing applications tailored for the UMPC platform,
but really, you can incorporate these optimizations into any application—you wouldn’t think of
writing a different application for each model of
laptop or desktop and I wouldn’t suggest you do
that for UMPCs. The UMPC optimizations that
add value usually tend to be ones that work fine
on a desktop or even add value to the usability
(especially if that desktop has a touch-enabled
display).
This is a great time to make one point clear: you
don’t need a UMPC device in order to develop applications to run on one. All the Tablet and Touch
Technology, speech, data synchronization, power
management, and network awareness APIs that relate to optimizing for the UMPC are “in the box” in
Windows Vista.
You do need a digitizer, either built-in to a Tablet PC
or UMPC or an external one for a desktop, in order
for the Tablet PC technology features of Windows
Vista to become more visible. Once Windows Vista
recognizes a digitizer is present, Windows Journal,
Tablet PC Input Panel, and handwriting recognition
features become available.
Optimizing an Existing Application:
UMPC Media Player Sample
You can do a few things to optimize your application for the small display size and touch-based digitizer of today’s UMPCs. The UMPC design guidelines cover this topic in detail, but since each application is unique you can choose what optimizations
work for you.
In the media player sample I’ll discus in this
section, I’ve chosen to create a C# Windows
Forms application that hosts the Windows Media Player control and I have optimized it for the
UMPC. You can find this sample application on
the Windows Vista development center for Mobile
PC (www.msdn.com/mobilepc) if you’d like to try it
out.
Adding a control to a Windows application and
calling the methods or properties on that control is
not specific to a mobile scenario, so I’ll focus on the
specific changes I’ve made to enable this application to take advantage of the display, digitizer, connectivity, Tablet and Touch Technology, and other
mobile factors in the UMPC.
Optimizations in this particular example range from
touch-friendly user interface, ink commands and
gestures, ink annotation, speech, power status, network awareness, and data synchronization.
Figure 5: Optimized user interface with full-screen capability.
Building the Optimized UMPC
User Interface
Adding Ink and Gesture components
An application user interface is completely dependent on the functionality of the application
and the style of the product. In my case I want
to provide a full-screen capable application that
has a clean UI and implements ink gestures and
speech features. I would certainly add additional
buttons and features for a shipping product, but
this sample just illustrates that you can get a lot
done with simple gestures.
Since the UMPC is running Windows Vista and the
Tablet PC features are built-in, you can add ink and
gesture functionality to an application without having
to redistribute additional dependencies.
Following the design guidelines, I’ve added larger
sized buttons and menus and have exposed the
main tasks that are great for this scenario: playing
media and navigating between tracks.
The handwriting recognition functionality provided
in Windows Vista has rich interfaces capable of creating and analyzing complex handwritten content, but
I am going to use just a small part of it here. I simply
want to recognize an individual character and perform an action mapped to it.
In Figure 5 you’ll notice that there is a “Full
Screen” button. This allows my application to
take advantage of the screen real estate and can
toggle back to windowed mode if the user chooses. You can see in the following code snippet how
the application toggles between the two display
states:
private void btn_FScrn(object sender, EventArgs e)
{
if (this.FormBorderStyle == FormBorderStyle.None)
{
this.FormBorderStyle = FormBorderStyle.Sizable;
this.WindowState = FormWindowState.Normal;
this.btn_fullscreen.Text = "Full Screen";
} else {
this.WindowState = FormWindowState.Maximized;
this.FormBorderStyle = FormBorderStyle.None;
this. btn_fullscreen.Text = "Windowed";
}
}}
My main point here is that you make sure you
scale your UI elements properly for touch displays
to give the user a great visual and input experience.
The touch screen fits really well with using ink-based
commands and gestures since they are a natural way
to interact (you can just point at items or write on
the screen).
To do this, I’ve chosen to use an InkOverlay to
collect the ink stroke and an InkAnalyzer object
to tell me what the character is recognized as. The
process is pretty straightforward as you can see in
Listing 1, I just add the reference to Microsoft.Ink,
declare an InkOverlay and InkAnalyzer and hook
up the events that I’m interested in. I want to be
Since the UMPC is
running Windows Vista and
the Tablet and Touch Technology
is built-in, you can add ink and
gesture functionality to
an application
without having to redistribute
additional dependencies.
23
Listing 1: Initialize the InkOverlay and InkAnalyzer objects
using Microsoft.Ink;
inkOverlay.SetGestureStatus(ApplicationGesture.Left,true);
// The Right gesture will skip ahead 20sec. in a media track
inkOverlay.SetGestureStatus(ApplicationGesture.Right, true);
// The ChevronLeft gesture will skip to the prev media track
inkOverlay.SetGestureStatus(ApplicationGesture.ChevronLeft,
true);
// The ChevronRight gesture will skip to the next media track
inkOverlay.SetGestureStatus(ApplicationGesture.ChevronRight,
true);
// The Tap gesture will stop/start the current media track
inkOverlay.SetGestureStatus(ApplicationGesture.Tap, true);
namespace WMPCSharp
{
public class Form1 : System.Windows.Forms.Form
{
private InkAnalyzer theInkAnalyzer;
private IinkOverlay inkOverlay;
private AxWMPLib.AxWindowsMediaPlayer Player;
public Form1()
{
// Initialize form components
InitializeComponent();
// Add a gesture event handler so you will be notified when
// a user creates one of the activated gestures
inkOverlay.Gesture += new
InkCollectorGestureEventHandler(inkOverlay_Gesture);
// Initialize the IinkOverlay with the Media Player control
// This will enable inking directly on the control
inkOverlay = new IinkOverlay(this.Player);
// Set the collection mode to accept both ink and gestures
// since you will look at both.
// Ink is analyzed as commands via the InkAnalyzer object
// Gestures will be mapped as actions in the Media Player
inkOverlay.CollectionMode = CollectionMode.InkAndGesture;
// Attach the IinkOverlay in front of the control. This
// enables the InkRenderer to render the ink on top of the ctl
inkOverlay.AttachMode = IinkOverlayAttachMode.InFront;
// Since the control has a black background color, change the
// ink color to a contrasting white
inkOverlay.DefaultDrawingAttributes.Color = Color.White;
// Add a stroke event handler so you will be notified when
// a user creates a stroke to be analyzed
inkOverlay.Stroke += new InkCollectorStrokeEventHandler
(inkOverlay_Stroke);
// Activate the following gestures in the IinkOverlay
// The Left gesture will skip back 20sec. in a media track
// Create an InkAnalyzer and associate it with the Ink object
theInkAnalyzer = new InkAnalyzer(inkOverlay.Ink, this);
// Finally, enable ink collection
inkOverlay.Enabled = true;
}
notified when each stroke is created so I can tell if
this is the letter I am interested in.
to be collected before analyzing if you want to support more elaborate commands:
For this simple example I am just watching for the
letter “m” and have mapped that to opening up the
music library of the Windows Media Player library.
You could add a timer to allow for multiple strokes
void inkOverlay_Stroke(object
sender,InkCollectorStrokeEventArgs e){
// Remove existing strokes and add the new one
inkAnalyzer.RemoveStrokes(theIA.RootNode.Strokes);
inkAnalyzer.AddStroke(e.Stroke);
// Analyze the ink
if (inkAnalyzer.Analyze().Successful)
switch (inkAnalyzer.GetRecognizedString())
{
case "m":
// Set the color
e.Stroke.DrawingAttributes.Color = Color.Green;
Player.Refresh();
openLibrary("music");// Perform the InkCommand
break;
When the user writes the letter “m” on the application (Figure 6), they will first see the white ink
stroke get created, and then it will turn green as
user feedback that the command was recognized
(Figure 7).
Figure 6: InkCommand associated with music library.
24
You can use this type of ink command throughout
your application. Once the music library opens in
this sample, it displays a list box with the current
music in the library (Figure 8). I’ve switched out the
standard list box with an ink-enabled list box from
the InkListBox sample on http://msdn.microsoft.com/
mobilepc. This gives me the opportunity to provide
consistent ink support on other controls. When the
user creates ink on the control, it automatically selects the song with a matching title. This functionality is built into the sample InkListBox control from
the developer center.
Once a user selects a song, it begins playing in the
main window with a selected visualization in the
background—similar to a standard media player.
The difference here is that the user can simply write
or tap directly on the window. In the sample, I have
mapped tapping the screen to toggle play/pause of
the media. If the user quickly moves their finger to
the left on the screen (a left gesture), the media will
skip backwards and replay the previous 20 or 30
seconds.
With an InkOverlay created, adding this gesture
support is equally easy as the Ink commands. In the
gesture event handler, the application needs to determine what gesture the user performed and to do
the matching task they have associated with it:
void theInkOverlay_Gesture(object sender,
InkCollectorGestureEventArgs e)
{
switch (e.Gestures[0].Id)
{
case ApplicationGesture.Tap:
if(Player.Ctlcontrols.get_isAvailable("Play"))
Player.Ctlcontrols.play();
else Player.Ctlcontrols.stop();
break;
case ApplicationGesture.Left:
SkipBackInCurrentTrack();
break;
}
}
Figure 7: Music library command recognized.
Figure 8: Ink-enabled list box.
The InkRenderer automatically renders the ink
directly on the visualization area of the media
player and updates the time (in this case decremented by 20 seconds) when the user creates the
“left” gesture (Figure 9). Your application could
add a host of these commands. Since discoverability can be difficult, be sure to add clear documentation, ToolTips, or alternative buttons for
these tasks.
Adding an Ink Annotation
Component
Adding tags to photos using ink is a great way to
organize photos on the go. Since you have already
enabled ink collection in your application and associated an InkAnalyzer, all the application needs to
do is to determine what to do with the recognized
text.
The simple case is to take all handwritten content
on a photo and add that text as keywords to the
Figure 9: User can use an ink gesture directly on the window.
image properties (the Windows Media Player control allows access to the Photo library as well as
music).
25
Listing 2: Adding support for Speech Commands in Windows Vista
using System.Speech.Synthesis;
using System.Speech.Recognition;
// Declare a Recognition Engine and Synthesizer for Speech
private static SpeechRecognitionEngine __SR;
private static SpeechSynthesizer __SS;
private void InitializeSpeech()
{
// First, create the main recognition context object.
__SR = new System.Speech.Recognition.SpeechRecognitionEngine();
__SS = new System.Speech.Synthesis.SpeechSynthesizer();
// Create simple grammar to trigger the Music library
GrammarBuilder gb = new GrammarBuilder("Sample",
SubsetMatchingMode.OrderedSubset);
gb.Append("Music");
gb.AppendDictation();
__SR.LoadGrammar(new Grammar(gb));
// Create simple grammar to play/pause the media
gb = new GrammarBuilder("Sample",
SubsetMatchingMode.OrderedSubset);
gb.Append("Play");
gb.AppendDictation();
__SR.LoadGrammar(new Grammar(gb));
__SS.Speak("Speech Recognition is Enabled");
// Enable Asynchronous recognition
__SR.RecognizeAsync();
}
// Speech Recognition Results
void sr_RecognizeCompleted(object sender,
RecognizeCompletedEventArgs e)
{
// Make sure there is something to process
if (e.Result != null)
{
string messageString = "Unknown command";
String[] ruleValue = e.Result.Text.Split(new char[] { ' ' });
// Speak out the command that was spoken
__SS.Speak(new Prompt(e.Result.Text + " Command received"));
switch (ruleValue[1].ToUpper())
{
case "MUSIC":
openLibrary("music"); // Perfom the command
break;
case "PLAY":
if (Player.Ctlcontrols.get_isAvailable("Play"))
Player.Ctlcontrols.play();
else Player.Ctlcontrols.stop();
break;
}
}
// Restart Asynchronous recognition
__SR.RecognizeAsync();
// This event is fired after the SR completed the recognition
__SR.RecognizeCompleted += new
EventHandler<RecognizeCompletedEventArgs>(sr_RecognizeCompleted);
// You need to tell the SR object which speech input device to
// use. You are going to ask it to use whatever was
// chosen in the Control Panel/Speech Recognition applet
__SR.SetInputToDefaultAudioDevice();
}
// Synthesize speech to notify the user of speech availability
In this sample (Figure 10) the InkAnalyzer recognizes the handwritten text and adds it to the list of
keywords for the photo. The analyzer renders the ink
on the image, but it has not altered the quality of that
image. You can use the ink as a means to obtain the
keywords, or the application can choose to keep the
ink—storing it in one of the binary properties of the
image data. This storage would allow the application
to render the ink on the photo when a user loads it
again and gives the user a chance to manipulate it.
You could extend this feature to recognize ink content in different ways. One addition could be to
interpret any content that is contained by a circle
as the photo title and any other
text as a set of keywords.
Choosing the right tool
for the task is key for mobile
applications and the UMPC is a
great platform to
base them on.
26
Adding a Speech
Recognition Component
In Listing 2 I have added support for speech recognition to
this sample. Speech commands
match up very closely to the
ink commands that I added earlier. The application
can use the same helper methods in the code for
both scenarios and can provide a consistent set of
ink and speech-related commands to make it more
intuitive for the user.
Adding good feedback for speech commands is
equally or more important than ink gestures since
there is no traditional interaction with the application. In this sample, the application will audibly
repeat the spoken command back to the user, but
your applications should use UI feedback such as
status messages as well.
Save the Batteries!
Pay Attention to Power
I have to confess that when I started writing Windows applications with the Windows 3.1 SDK, I really didn’t pay any attention to power management
until a few short years ago. Why? Until developers
started seeing battery operated devices, there really
was no benefit in making sure an application did
anything different with power.
Now, almost every conversation I have about portable devices starts off with the question: “How long
does the battery last?”
Optimizing an application’s usage of power is difficult to measure, but can have a lasting impact
on the user. There have been a set of Win32 APIs
around for a long time that allow you to determine
if you are plugged in to AC current or on batterypowered DC. You can also determine what the level
of the devices battery is via these APIs.
Utilizing this information, an application can now
determine whether to enable high intensity CPU/
GPU features like animations or shading—or give
the user a choice to disable those features.
A great example is when you are starting a movie
while on battery power. If the application knows the
length of the movie is two hours, but the battery level
shows less than that amount is available, it could notify the user of their options: watch the movie at current settings and risk not finishing or change screen
brightness to have a better chance of completing.
To query the power status in your application, you
can get most of what you need from the GetSystemPowerStatus API:
[DllImport("Kernel32.DLL",
CharSet = CharSet.Auto, SetLastError = true)]
private extern static
bool GetSystemPowerStatus(
SYSTEM_POWER_STATUS SystemPowerStatus);
It returns a SYSTEM_POWER_STATUS class containing all the relevant information about AC and
battery connections:
[StructLayout(LayoutKind.Sequential)]
public class SYSTEM_POWER_STATUS
{
public byte ACLineStatus;
public byte BatteryFlag;
public byte BatteryLifePercent;
public byte Reserved1;
public Int32 BatteryLifetime;
public Int32 BatteryFullLifetime;
}
The ACLineStatus tells you whether the machine is
plugged into an outlet, and the BatteryFlag informs
you of the current state of the battery:
public enum ACLineStatus : byte
{
Battery = 0,
AC = 1,
Unknown = 255
}
public enum BatteryFlag : byte
{
High = 1,
Low = 2,
Critical = 4,
Charging = 8,
NoSystemBattery = 128,
Figure 10: Adding tags to photos with ink annotations.
Unknown = 255
}
The rest of the information regarding power that
your application will be interested in monitoring is
the WM_POWERBROADCAST message. To monitor this, call the RegisterPowerSettingNotification
method to request notification of the following
power management events. These events are sent
with the WM_POWERBROADCAST (http://msdn2.microsoft.com/en-gb/library/aa372715.aspx) message:
• PBT_POWERSETTINGCHANGE PBT_APMBATTERYLOW
• PBT_APMPOWERSTATUSCHANGE
Optimizing for Network Connectivity
I previously mentioned that UMPCs have the latest
network connectivity available, including wireless
LAN (Wi-Fi 802.11b/g), and also have the capability
to connect to high-speed CDMA or 3G cellular-based
networks (those provided by your cell phone carrier).
This is a great advance for mobile computing scenarios; having the ability to get to my data, media, and games on the go is great. But just like my
desktop and its broadband Internet that I can’t live
without, I want my mobile PC to be constantly connected too. I can’t even imagine what life would
be like without broadband Internet access anymore
on my desktop. The mobile computing possibilities
are just starting to unfold with an always-on con-
Now, almost every
conversation I have about
portable devices starts off
with the question:
“How long does the battery last?”
27
nection wherever we are whether it is the now traditional Wi-Fi or a new GPRS connection.
Am I Online or Offline?
Data Synchronization Is a Must
“FileNotFoundException has
occurred”
Inevitably, users will need to update, upload, or
synchronize their data. One approach to this type
of data management, used by many ISVs today, assumes that the user is always online and has a server available to communicate with (either Wi-Fi or
WWLAN). It’s not difficult to see the shortcomings
in this solution when a connection is sporadic or
the user travels outside the range of the network.
In building applications in these scenarios, you
might think that you need to do nothing—if the network is available, via whatever means, your application simply sends or receives data and the particular
network stack is completely transparent.
Sure, that statement is partially true. But think
about all the applications that you’ve used in the
past that would hang before they figured out that
a FileWrite or WebRequest had timed out and the
network wasn’t available. Not a great user experience.
All the application really had to do was add some
simple code to see if the network was active before
making the request. Your application should do the
same and use good notifications to your user to let
them know what network state changes are occurring.
Save Your Energy;
Reduce the Chattiness
Beyond the connection issue is connection speed
and how you can optimize your application regarding it. Similar to being smart about detecting the existence of a network, your application
can easily determine the connection type and
throughput.
Using the Network Location Awareness (NLA) service in Windows Vista, your application can query
current network properties and be notified of any
changes in those networks. Depending on the situation, you can optimize network communication for
the user or let them select from options.
For a client/server relationship you could automatically reduce the number of times the client talks to
the server or choose to reduce the amount of content delivered; omitting images for example.
You can use the EnumNetworks method in INetworkListManager to enumerate the network profiles for Windows Vista.
Each network profile has a name and description
that you can access by using the INetwork::GetName and INetwork::GetDescription methods.
You can use the INetwork::GetState method to determine whether the network is connected to the
computer and whether the connection has Internet
connectivity. Use the INetwork::GetType method
to determine whether the network is a managed
(domain) network and whether the user is authenticated on that network.
28
But what is a low-cost alternative? The Data Synchronization service in Windows Vista is one. The
simple summary for Windows Vista Data Synchronization is that it allows you to reliably write data to
local files and that the Sync Center will update the
cache periodically when a connection is available.
This Data Synchronization service was available in
Windows XP but Microsoft updated it for Windows
Vista. The Sync Center has a new UI, is integrated
into the shell, and has an integrated conflict resolution capability and updated API.
This service provides the framework and some UI
for your needs, but does still require you to determine what to synchronize and how to handle conflicts in files—especially important if you choose to
allow data to be modified on both the remote and
synchronized locations.
If I want to automatically synch the data without
the user having to initiate, or if I want to use my
own UI for the sync, I can still do this with the Sync
Center API.
Whether you use this Data Synchronization service, the focus here is to enable your applications to
seamlessly handle all connectivity situations, even
sometimes-online ones by using the rich set of APIs
available to you.
Summary
Creating an application that is targeted for the
UMPC is potentially no different than writing a robust Windows Vista application. Choosing the right
tool for the task is key for mobile applications and
the UMPC is a great platform to base them on.
Software that is well targeted for mobile scenarios
and adds clear value to users is what will really help
define where these mobile devices go in the future.
I look forward to seeing upcoming Windows Vista
software and UMPC hardware so I can leave my
notebook on my office desk when I’m on the go.
Todd Landstad
Windows Vista Mobility:
Why Should You Care?
“Why should I upgrade to Windows Vista™?” is the single mostasked question I get at work these days. Curiosity is driving a lot of
folks towards this latest and greatest operating system from Microsoft. What’s new?
What does it do? What’s in it for me? Everyone wants to know what’s so special about
it and if it is worth it. “How much time do you have right now?” is usually my first
reply, closely followed by, “Do you have a laptop?”
O
f course many professionals and knowledge
trusty companion. In this article, I’ll explore Winworkers do have a laptop, and that often indows Vista and its new mobile PC-related features,
cludes application developers too. Windows
be they from the end-user point of view, or from
Vista offers tons of new features and enhancements,
the developer angle looking to exploit the power of
from the new spiffy hardware-accelerated user interWindows Vista for mobile applications.
face with Aero to all the builtin .NET Framework 3.0 techFast Facts
nologies (WPF, WCF and WF),
First Look: Popping
Windows Vista—like its
Microsoft® Windows® Superthe Hood Open…
Redmond-born predecessors—
Fetch™ to Windows ReadyBoost™, Microsoft packed a
has two stories to tell: an
lot of power under the hood
Windows Vista Control Panel
end-user story, and a developer
(see sidebar, Windows Vista:
sports many options, all orstory. From the perspectives
Beyond Mobility). But throw
ganized by category. You will
of the mobile PC user and the
Windows Vista on a mobile PC
find old classics in there like
mobile PC developer, Windows
and get ready for some serious
the ability to adjust display
Vista provides many chapters
improvements for the mobile
settings and uninstalling prowith bold tales of enhanced
professional. The term mobile
grams, but you will also notice
PC describes all portable comnew options, like a dedicated
ink support, personalized
puters such as Ultra-Mobile PC,
security section and a whole
recognition, touch input,
notebook, laptop and Tablet
new section just for mobile PC
better data synchronization,
PC running the full Microsoft
settings. Figure 1 illustrates the
new programmatic interfaces
Windows operating system.
various sub-sections related to
for power and network
mobile PC settings, which inawareness, and more.
Following the release of Miclude the following:
crosoft Windows XP Professional in 2001, which also saw the launch of the
• Windows Mobility Center
new Tablet PCs, Microsoft went back to work to
• Power Options
bring mobile PCs to the next level. Today, Win• Personalization
dows Vista pushes the envelope on mobility over
• Tablet PC Settings
its predecessor and it shows. Where Windows XP
• Pen and Input Devices
introduced us to the Tablet PC features of using pen
• Windows Sync Center
computing and electromagnetic digitizers; Windows
• Windows Mobile Device Center
Vista extends the Tablet PC experience with support
for touch. Where Windows XP featured a standard
recognizer that could decipher your handwriting
Windows Sync Center
and convert it into text fairly accurately, Windows
Vista can learn the quirks of your penmanship (or
allows you to manage the
lack thereof) using a personalized recognizer and
synchronization of all your
automatic learning.
You do not have to explore very far to notice all the
mobility features integrated in Windows Vista. Simply pop open the mobile PC options in the Windows
Vista Control Panel, and you’ll discover a plethora
of goodies enough to make your notebook smile,
be it a brand new machine or your newly upgraded
data and files between various
computers, devices,
external storage,
and network servers.
Nickolas Landry, MVP
[email protected]
Nickolas Landry is a senior .NET
architect and Enterprise Mobility
Practice Manager in New York
for Infusion Development, a
Microsoft Certified Gold Partner
which offers quality software
development services, developer
training and consulting services
for large financial firms in the
New York, Boston, and Toronto
areas.
Known for his dynamic and
engaging style, he is a frequent
speaker at major software
development conferences
worldwide like Tech Ed and
MEDC. He’s a member of the
MSDN Canada Speakers Bureau,
a Microsoft MVP on Device
Application Development, and
is the Vice-President of IASA
New York (www.iasahome.org).
Aside from his work in designing
business solution architectures
using .NET technologies, Nick
provides mentoring services
in architecture, design, and
.NET development, authors
and teaches .NET classes,
performs system audits and
business analyses, and profiles
technologies for various
enterprise scenarios.
Nick specializes in .NET mobility,
OOP & SOA, architecture and
design patterns, and application
security. He has written
about mobile development for
magazines and has helped to
develop several .NET mobility
courses for Microsoft, has been
a technical editor for many
book titles, and holds several
professional certifications from
Microsoft and IBM. Blog: http://
activenick.infusionblogs.com
Windows Vista Mobility: Why Should You Care?
29
Figure 1: The new Mobile PC section in Windows Vista Control Panel provides everything you need to
manage your laptop or Tablet PC, from power settings and connecting to a projector, to customizing the
tablet pen and connecting to Windows Mobile devices.
I cover most of these features over the next few
pages, and for many of them, you get the chance to
dig deeper into these mobility topics by reading the
subsequent articles included in this special CoDe
Focus issue.
Windows Vista:
Beyond Mobility
Windows Vista offers a lot beyond
the mobility features covered
here, including an enhanced
and redesigned security model,
new identity management with
Windows CardSpace, hundreds
of productivity features, new
Windows Vista APIs, Microsoft
.NET Framework version 3.0
built-in, DirectX 10, Windows
ReadyBoost, Windows
SuperFetch, Internet Explorer 7,
Windows Media Player 11,
a built-in desktop search engine,
and much more.
Getting Started with Windows
Mobility Center
Windows Mobility Center presents you with many
useful options related to the day-to-day use of a
mobile PC (Figure 2). Each pane in this window
provides you with a quick setting you can adjust
with a single click, including trivial but essential
features like adjusting your screen brightness level
or adjusting or muting the volume control. You can
click an icon like a button in each pane, allowing
you to customize your settings to your liking. You
will also find interesting settings that were normally only accessible by jumping through Windows
XP hoops, or entirely new features, including the
following:
To learn more about Windows
Vista and why you should develop
on and for it, visit Windows Vista
Developer Center at http://msdn.
microsoft.com/windowsvista.
30
• Adjust your immediate Display brightness,
or click the icon to change your Display,
Sleep (discussed below), and Brightness settings whether you’re running on battery or AC
power.
• Adjust the Volume, mute the sound, or use the
icon to access your sound device properties.
• Change your power scheme, also known as
your Battery Status, to the standard settings
like Balanced, Power saver or High performance. The icon allows you to configure each
of those power plans.
• Turn your Wireless Networking (i.e., Wi-Fi)
access on or off. The icon pops open the Con-
nect to a network dialog box that shows available Wi-Fi access points.
• Provided that your mobile PC has the tablet feature (as a slate or convertible laptop),
a Screen Orientation option is available to
easily rotate the screen between portrait and
landscape modes. Unfortunately Windows
Vista still does not remember separate desktop icon position sets for portrait and landscape modes. The Panel icon provides access
to the Tablet PC Settings dialog box.
• If you’re like me, you connect your laptop to
a lot of external monitors and projectors. Windows Mobility Center features a button to easily connect to an External Display, or use the
icon to access the Display Settings.
• The new Windows Sync Center, accessible
here, allows you to manage the synchronization of all your data and files between various
computers, devices, external storage, and network servers. I will cover this setting in greater
detail below.
• You also have access to a one-click Presentation Settings toggle that lets you enter
Presentation mode where you can enable or
disable notifications and screen saver, adjust
your volume, or temporarily change your
Windows Desktop background image. Use
the associated icon to configure these settings.
Use Windows Vista Mobility Center to get access
to the most often used mobile PC functions. Finally, users won’t suffer from any fumbling around to
locate the proper mobility option when they’re on
the go.
Power Management
Have you ever experienced the “hot-laptop-in-thebag” syndrome? I’m sure you did at least once.
Let me refresh your memory. You prepare to leave
work, heading to the airport on a business trip.
You put your laptop on Standby, drop it in your
bag, and off you go. Later, as you sit not-so-comfortably in your middle seat between the Queen of
Chatter and the King of Snooze, you take comfort
in knowing that you’ll have your laptop along for
the ride to get a little work done, catch-up on email, listen to some music, or even watch a movie
during your cross-continental trek. You reach for
your laptop in your bag and… the horror! Your
laptop is burning hot, and it’s still on!!! Fearing
the worst, you open the lid and the power indicator taunts you with… 12% battery left. You’re
doomed. You’re now stuck watching a really mindless movie using rented headphones and reading
the airline magazine for six hours. And you can
swear you can detect a grin on Mr. McSleepy’s face
next to you.
Windows XP provided you with several options
when it came to suspending your work on a laptop.
Closing the lid would typically put the machine
in Standby mode, where most of the hardware is
turned off and your work session is preserved in
memory, consuming just enough power to keep it
alive for days. As your battery would drain very
slowly in Standby mode, after a while, Windows
XP should briefly wake up to transition to Hibernate mode, whereby it transfers the contents of
your system memory to a special Hibernate file
on disk and then turns your machine off entirely.
Unfortunately the Hibernate mode was not always
reliable, as it required a minimum amount of system resources to complete the API call. If you had
too many things loaded at once in your Windows
session, a symptom that occurs more often if you
have a lot of RAM in your system (usually 1.5 GB
or more), the Hibernate call would fail, and Windows XP would remain powered on without trying
to go back to Standby mode again. If you leave
the laptop running in a tight bag with no ventilation long enough, you’ll end up with a burning hot
paper weight.
Figure 2: The new Windows Mobility Center, accessible via the Mobile PC section of Windows Vista
Control Panel, provides quick adjustments to various mobility features. Clicking on each pane’s respective
icon opens a settings window where you can customize how each of these features operates.
Windows Vista solves this issue with a new power
state called Sleep. This new sleep mode combines
the best of both worlds: fast resumes like the old
Standby mode and battery-efficient disk persistence
like Hibernate. Sleep handles both and does not
need to switch between modes as it can transition
your session state from memory to disk, thus avoiding the need to “wake-up” from Standby before hibernating.
Baking Power Awareness in your
Mobile PC Applications
While users can monitor the battery level on their
mobile PC and manage their power management
settings on their own, a good smart client application should be made power-aware. That is,
your applications should know about the current
power status in order to adapt their behavior depending on the power mode. If the user switches
to a power saver mode to maximize their laptop
battery, your application should behave properly
and refrain from initiating resource-intensive operations, deferring them to a later time when the
mobile PC is running on AC power or is running
in high performance mode. How can your application detect this power state change? If the
mobile PC is put to sleep, how should your application react?
Neil Roodyn wrote about power management on
Microsoft Windows XP Tablet PC Edition in the
previous issue of CoDe Focus on Tablet PC &
Mobile PC development (see Power to the People, http://www.code-magazine.com, Quick ID
0512102). For a Windows Vista application, you
can use new features of the Power Management
SDK, which is a native code API. Basic power notifications are also available in the .NET Framework via managed code. Unfortunately, any more
on this topic is beyond the scope of this article.
For more information, see “Power Management”
in the Windows SDK.
Figure 3: Tablet PC Input Panel in Windows Vista will look very familiar to Windows XP Tablet PC Edition
users. However, Windows Vista has the ability to learn your handwriting as you correct what is being
recognized. The more you use it, the more it learns.
Tablet and Ink Support
The top feature I had desperately been waiting to
use on my Tablet PC is the enhanced ink support
in Windows Vista. Don’t get me wrong: Windows
XP’s handwriting recognition and ink handling
were pure genius. It was so good that most people assumed I had been training it to the specifics of my lousy handwriting beforehand. It turns
out most people had no idea you simply could not
train Windows XP to enhance ink-to-text recognition results. On the other hand, Windows Vista
can learn, and it does so in a very smart way.
Figure 3 shows that Tablet PC Input Panel in Windows Vista has not changed much visually compared
to the Windows XP Tablet PC Edition version, but
Microsoft has dramatically improved the ink recognition engine. For starters, Windows Vista does a
good job recognizing things you cannot necessarily
lookup in a dictionary, such as e-mail addresses and
URLs. Windows Vista also allows you to use pen
gestures, or “flicks” to perform simple commands
and actions.
Want to Hear More
About Windows Vista
Mobility?
If you want to hear more about
Windows Vista mobility and
learn directly from the Microsoft
product teams and Microsoft
MVPs, join me and thousands of
other developers and dig into the
latest development technologies
for mobile PC and Windows
Mobile at the following events:
* Microsoft MEDC 2007, May
1-3 2007, Las Vegas, NV
(www.medc2007.com)
* Microsoft TechEd 2007,
June 4-8 2007, Orlando, FL
(www.microsoft.com/events/
teched2007)
* Mobile Connections Fall 2007,
November 5-9 2007,
Las Vegas, NV
(www.devconnections.com)
“You should have been a doctor…”
If you have ever been told that your handwriting is
so illegible that you would have made a fine doctor,
I’m sure you may have your doubts when it comes to
31
handwriting recognition using ink-to-text on a Tablet PC. Since Windows XP could not “learn” your
handwriting or be trained to recognize it, the recognizer would often make the same mistakes. Adding a
new word to the built-in dictionary would often yield
better recognition results, but when a quirk of your
handwriting makes it easy to confuse one word for
another, Windows XP often seemed to get it wrong.
Where Is the
Windows Vista Tablet
PC Edition?
Note that Windows Vista does
not require a separate Tablet PC
edition anymore. All editions of
Windows Vista already include
Windows Tablet PC technology,
whether you have a tablet
digitizer on your mobile PC or
not. This new integration now
allows all Windows Vista users
to manipulate tablet ink. For
example, if you receive ink data
in an e-mail from a Tablet PC
user, you can convert this ink to
text on your standard Windows
Vista desktop or laptop.
For example, when signing e-mails with my “ActiveNick” nickname, Windows XP would usually
confuse my “v” for an “r” and always suggest “ActireNick”. By clicking that word in the Tablet PC
Input Panel, I could look up the other probable
results, and sure enough, “ActiveNick” would be
the second or third choice. With Windows Vista,
I first had the same problem, but by choosing
the properly recognized word in the probability table, the personalized recognizer actually
“learns” that my lowercase v’s look more like
lowercase r’s, but it’s still a “v”. I correct Windows Vista once, and now I don’t worry about
it anymore. In a nutshell, the more you use ink
recognition in Windows Vista, the more it learns
and the more accurate it gets at recognizing
your handwriting. This personalized recognizer
tracks handwriting data separately for each user
on the mobile PC, and you can turn it off if you
so desire.
Touched for the Very First Time
I always found it funny when people would try my
Tablet PC for the first time. Maybe it’s because I
use an LE1600 slate tablet from Motion Computing
that has no keyboard, but people would always try
to avoid touching the screen when writing with the
stylus. Of course, this cannot feel natural; we always
rest the side of our hand when writing with pen and
paper, and that’s why Tablet PCs relied on electromagnetic digitizers for pen computing on Windows
XP. These digitizers use electromagnetic signals
emitted around the screen to detect a passive sensor
(i.e., no battery required) in the stylus and extrapolate the stylus position to drive the mouse cursor on
screen. The underlying principle is, in fact, not entirely dissimilar to how Global Positioning Systems
(GPS) work when triangulating positions on Earth using
satellite signals. This technique
makes it possible to track the
In a nutshell, the more
stylus tip when simply hoveryou use ink recognition in
ing above the screen without
touching it.
Windows Vista, the more it learns
and the more accurate it gets at
recognizing your handwriting.
But a lot of people would try
to push buttons with their fingers. It’s only natural; you do
it all the time when you withdraw money at most ATMs,
use the self-service pumps at the gas station, or
use the self-check-in terminals at the airport.
Windows Vista adds support for tactile input using touch screens. With touch input, a user can
still use a stylus but only as a tactile device, just
32
like on a Windows Mobile Pocket PC. Touch digitizers add flexibility for the user, but are less precise than electromagnetic digitizers. This means
your handwriting will look less natural and more
“broken” with jagged edges, sort of like digital
delivery pads or credit card terminals that rely on
digital signatures. And because touch input does
not involve electromagnetic tracking, Windows
cannot detect the “hover” of your stylus over
the screen. Note that Windows Vista also allows
for devices with a hybrid touch/electromagnetic
digitizer. Such devices would typically disable the
touch mode when the hardware detects the stylus in range, leaving the touch support for finger
input only.
The Ink Developer Story
CoDe Magazine has already had a few articles
about adding ink support to a .NET Framework
application, starting with the article at Quick ID
0512032. The new 1.7.5 release of the Tablet PC
Platform SDK extends the base functionality with
a new InkAnalysis API, which is supported by the
new version of the pen input subsystem in Windows Vista. Markus Egger introduced ink analysis in a previous CoDe Focus article (Quick ID
0512042).
You can also use ink with Windows Presentation
Foundation (WPF), as discussed in this issue in,
“Basics of Ink in Windows Presentation Foundation,” by Billy Hollis. Microsoft designed WPF to
support ink input, treating the stylus as the mouse
pointer or as true pen input.
To learn more about adapting your mobile PC applications using a touch interface, make sure to read
Hilton Locke’s article, “Introduction to Windows
Touch Technology in Windows Vista,” in this issue
of CoDe Focus.
All these new additions for developers, combined
with the enhanced learning and personalization
features for ink in Windows Vista, greatly improve
on your application’s abilities to process handwriting as a viable alternative to keyboard input in enterprise scenarios running on mobile PCs.
Building Network-Aware Applications
Talking about mobility, I use the term “smart client” a lot, which usually prompts the question
“What’s the difference between a rich client and
a smart client?” Power management—as described
above—is part of that answer. But the first and
foremost attribute of a smart client is its ability to
work both in online and offline modes, completely
transparently from user interaction. If a user wants
to save their data, the last thing they want to hear
is that no connection to the server is available. It’s
up to you as a mobility developer to teach your
applications to think and to be aware of their environment.
Neil Roodyn discussed network awareness in the
article, Know your Networks (Quick ID 0512092),
in the previous issue of CoDe Focus. Now thanks to
the new Network Awareness API, Windows Vista
applications can be informed of changes in network
configuration. With this API, your application can
now find answers to many network-related questions, such as:
• Am I connected to a network?
• What type of network is it and what are its attributes?
• How fast is this network?
• Is this an intranet network and do I have Internet access?
• Can I trust this network?
• What should I do when I lose connection to
the network? What about when I get reconnected?
• Are there other alternative networks I can use
based on predefined criteria?
• A new network was just added. Should I use
it? Does it match my needs?
Whereas Windows XP only exposed the Network
Location Awareness (NLA) service via the Windows Sockets 2 API, the new Network Awareness
API in Windows Vista is a native Win32 API that
relies on COM. Developers will find a subset of
the Network Awareness API in the form of managed classes for managed development. See the
NetworkInterface class in the System.Net.NetworkInformation namespace for more information. You can also use the NetworkChange class
in the same namespace to receive notification of
network changes in your application.
The specifics of the NLA service and this API are
beyond the scope of this article. You can read more
about NLA and the Network Awareness API by
looking up “Network Location Awareness” in the
Windows SDK.
Managing Files with Sync Center
When Bill Gates founded Microsoft in 1975, his vision was to bring “a computer on every desk and
in every home.” Nowadays, average folks, and especially information workers, have a multitude of
computers and devices, including computers at
work, at home, at the ski cottage, laptops, Tablet
PCs, Pocket PCs, Smartphones, Ultra-Mobile PCs,
and more. Übergeeks and power users often have
all of these. And while we live in a world of information at our fingertips, that information lives all
over the place. These various computers have created islands of data that are very hard to manage
and keep in sync, often resulting in having multiple
copies and/or versions of the same documents all
over the place. Finding the latest version has become a nightmare.
Enter Windows Sync Center, a new centralized
console in Windows Vista where you can manage
your files and synchronize them across multiple
Figure 4: Windows Sync Center provides a central interface to manage sync partnerships with other
devices, computers, external storage devices, and mobile devices. It can even manage and monitor SQL
Server synchronization jobs.
PCs, network servers, and mobile devices. Windows
Sync Center allows you to manage the synchronization of files residing on your Windows Vista machine and on other sources, including:
• Other computers, using a PC-to-PC sync
• Network files from other computers or servers
you need access to when offline
• Removable storage like USB keys and flash
memory cards (e.g., Compact Flash, SD, Memory Stick, etc.)
• Mobile devices
You’ll need to set up a sync partnership before you
can synchronize files between sources. Figure 4
shows how Windows Sync Center facilitates the
management of such partnerships. From this interface, you can initiate
syncs, monitor syncs
in progress, cancel
syncs, and review
The first and foremost
potential issues that
attribute of a smart client is its
might occur during
synchronization.
ability to work both in online and
offline modes.
From a developer’s
point of view, you
can use Windows
Sync Center since it
can manage the synchronization of data with Microsoft SQL Server™ databases. This means that
SQL Server 2000 or 2005 exposes a sync handler
interface that Windows Sync Center can monitor
and manage. You can also interact with Windows
Sync Center programmatically. Windows Vista
provides APIs to create your own sync handlers
that leverage the services of Windows Sync Center to synchronize files and data from your own
applications. This even allows you to define your
own property sheets to extend the Windows Vista
33
dard Windows Sync Center user interface. However, creating sync handlers for sync center goes beyond the scope of this article. You can refer to the
Windows Vista Developer Center at http://msdn.
microsoft.com/windowsvista for more information
on specific APIs for Windows Vista.
Windows Mobile Device Center
The new Microsoft Windows Mobile® Device
Center (WMDC) sits on top of the Windows Vista Sync Center to manage partnerships, connectivity, and synchronization with Windows Mobile
2003 and Windows Mobile 5.0 devices. You can
download WMDC at http://www.microsoft.com/
windowsmobile/devicecenter.mspx
Figure 5: Windows Mobile Device Center in Windows Vista replaces ActiveSync for the management of
partnerships and connectivity with Windows Mobile devices. You can use it to set up a partnership with a
new device or simply connect to a device in “guest” mode.
user interface where users would configure syncspecific settings associated with your sync handler.
Your code can also use the Windows Sync Center
API to generate results and provide feedback on
synchronization by posting to the Conflict Store
and the Event Store to make sure your application
can inform the user on the sync status via the stan-
In essence, WMDC replaces Microsoft ActiveSync® as a core Windows component managed by
Windows Sync Center. Once you have installed
WMDC, connect your Windows Mobile device.
WMDC greets you with a Welcome screen as
shown in Figure 5 and offers you a choice between setting up a device on your computer,
which creates a new partnership with this device,
or simply connects the device with no setup, similarly to ActiveSync’s “guest” mode.
If you choose to set up a partnership, WMDC
will ask you to pick the item types to synchronize as part of this partnership (Figure 6). Your
options include contacts, calendar items, e-mail
messages, tasks, notes, mobile favorites, files, and
Microsoft OneNote® items. Note that for contacts, calendar, e-mail and tasks, you can elect
to sync directly over the air with Microsoft Exchange Server and Direct Push, but you cannot
sync any of these with both the desktop and the
server. If a server option is available in your company, you should use it as your desktop should
be in sync with your Microsoft Exchange Server
as well.
Once you’ve set up the partnership, WMDC will
ask you to give it a name, and it will then proceed
to sync your new device with your Windows Vista
machine using Windows Sync Center (Figure 7).
If an error or conflict occurs, or if you want to review the synchronization results, open Windows
Sync Center and click on the View sync conflicts
and View sync results tasks in the left pane to review errors, warnings, and other sync information.
Also note how your WMDC partnership is now
Figure 6: Windows Mobile Device Center allows you to keep many item types in sync between your
computer and your device, including contacts, calendar items, e-mail messages, tasks, notes, mobile
favorites, files, and OneNote items. Note that for contacts, calendar, e-mail and tasks, you can elect to
sync directly over the air with Microsoft Exchange Server and Direct Push.
34
In essence, Windows
Mobile Device Center replaces
ActiveSync as a core Windows
component managed
by Windows Sync Center.
visible when you click on the View sync partnerships task. You can create multiple partnerships to
all your different devices from the same Windows
Vista machine.
With the partnership set up, WMDC and Windows
Sync Center kick into action every time you connect your device to your Windows Vista machine,
or whenever a data change occurs on either side of
the sync partnership. The WMDC window (Figure
8) is accessible via Mobile PC Control Panel, and
provides many options:
• Get updates for your device.
• Add and remove programs on your device.
• Add media files to your device, or import media files from your device to your Windows
Vista machine.
• Browse the contents of your device’s storage
space, including storage cards. Note that device storage is also accessible from Windows
Explorer as a regular Windows drive.
• Change the content sync settings.
• Manage the partnership, such as how to handle conflicts.
• Adjust the connectivity settings. WMDC can
connect with devices via USB, infrared, and
Bluetooth.
Figure 7: Windows Mobile Device Center maintains your data items in sync between your Windows Vista
machine and your Windows Mobile device. Sync Center itself handles the synchronization tasks.
Where to Go from Here
I have only scratched the surface of the features
that Windows Vista supports for mobile PCs. The
true power of Windows Vista lies not in its new
Control Panel options, its new looks, or new settings; the true power of Windows Vista rests in your
hands as a developer of applications using the .NET
Framework. As you discover the new APIs, SDKs,
and services available to your applications, you will
realize how—just like its other Microsoft Windows
predecessors—Windows Vista is first and foremost a
premier development platform, and its best features
have not been written yet. It’s up to you now. Keep
reading through the pages of this special CoDe Focus issue, learn about Windows Vista programming
for the mobile PC, and you’ll finally have the answer
for you and for your users: “Yes, Windows Vista is
definitely worth it!”
Nickolas Landry
Figure 8: Windows Mobile Device Center can also install and remove programs on your Windows Mobile
device, add and import media files to and from your device, browse the contents of your device’s storage,
including storage cards, and adjust the partnership and connection settings.
35
The Proper Developer
Environments for Mobile PC,
Tablet PC, and Ultra-Mobile
PC Applications
Eliot Graff
[email protected]
(425) 703-9870
Eliot Graff is the Lead for the
Developer Experience Team
within the Mobile and Tailored
Platforms Division at Microsoft.
He started programming in the
late 70s, using punch cards and
FORTRAN to solve biomedical
engineering problems. A former
English professor (obviously
not very good at FORTRAN), he
mostly spends his time chasing
his two children and recuperating
with his wife. His latest hobbies
given to him by his children
include dinosaurs, Spiderman,
and hip-hop.
When Microsoft® Windows® XP Tablet PC Edition appeared in
2002, developers were sometimes confused about whether to
write code on a Tablet PC or if it were possible to develop Tablet
PC applications on existing desktop computers.
The solution turned out to be fairly straightforward, if less than intuitive. However,
with the addition of Ultra-Mobile PCs (UMPCs), multiple releases of the Tablet and
Touch Technology API, and now the Windows Vista™ operating system, this question
arises again: What do you need to do to set up your developer environment and
create applications for these form factors?
W
ith the increased profile and importance
Tablet PC and UMPC—run Windows XP or Winof mobile PC considerations—network
dows Vista. You develop for them by drawing upon
awareness, power management, auxiliary
the .NET Framework and Windows API. Mobile
displays, screen resolutions, Tablet and Touch
PCs are full computers, although they are often
Technology, and others, developing applications
confused with Windows Mobile® devices. For
for mobile customers has
Windows Mobile devices, you
never been more dynamic,
use the .NET Compact FrameFast Facts
exciting, and simultaneously
work for development and run
The Recognizer Pack
challenging. Your customers
Windows Embedded on the
provides ten handwritingare probably using laptops,
client.
notebooks, Tablet PCs, Ulrecognition engine languages.
tra-Mobile PCs, or a combiA quick note: you can find all
nation of these. Do you apof the resources mentioned in
proach these form factors as devices—requiring
this article through the Mobile PC Developer Cenemulators—full-fledged Windows computers,
ter, the main Microsoft portal for information about
or some sort of hybrid? The truth is that it dedeveloping applications for mobile PCs (http://
pends. It depends on which version of Windows
msdn.microsoft.com).
you’re developing on, which version of Windows
you’re developing for, which version of the various SDKs you’re using, and which form factor
Windows Vista
your applications deploy on. But before you fling
your UMPC through a twelfth-story window,
With Windows Vista, one of the major changes
take a breath. It’s really not all that complicated,
for developers of Tablet PC applications is that the
and it’s getting simpler and simpler.
Tablet PC DLLs—and hence the Tablet and Touch
Technology—ships as a part of Windows Vista
Home Premium, Windows Vista Enterprise, WinSize Matters
dows Vista Business, and Windows Vista Ultimate
editions. If your development computer has one of
Well, at least the size of the
these versions installed, there’s no need to downDo you treat these form
operating system’s footprint.
load any additional packs. Ink, recognition, and
Mobile PC form factors are
Tablet and Touch Technology are sitting on your
factors as devices,
computers, running the full
computer—even if it is not a Tablet PC—and will
requiring emulators,
Windows operating system. I
just start working when called upon. Likewise for
full-fledged Windows computers, cannot emphasize this enough, your customers’ computers: if end users are running
because this is the main point
Windows Vista, all Tablet and Touch Technology is
or some sort of hybrid?
of confusion developers run
available for those running these premium versions,
into. Mobile PCs—including
as well.
38
The Proper Developer Environments for Mobile PC, Tablet PC, and Ultra-Mobile PC Applications
To develop your applications, you must install Windows Software Development Kit for Windows Vista
and .NET Framework 3.0 Runtime Components.
You’ll find this single download is available on
MSDN or through the Windows Vista Developer
Center. The Windows Software Development Kit
(SDK) includes the necessary materials to develop
applications with, among other things:
• The .NET Framework 3.0, including Windows
Presentation Foundation (WPF).
• Tablet PC features that enable input, recognition, and rendering of ink.
• User experience guidelines for developers of
mobile applications.
• Improved APIs for handling changing network
conditions.
• Windows SideShow™
• Ink Analysis (Note: When you add a reference to one of the InkAnalysis namespaces,
you must also register the corresponding DLL
in the GAC. You can either do so manually
or also add a reference to IALoad.dll, which
handles registration for you.)
• Data synchronization so that customers have
the most current information in the least obtrusive manner.
• Network awareness APIs to handle connected/disconnected scenarios.
• Power management functions to keep batteries
operating for the maximum amount of time.
• Many samples.
• Connections to communities from which you
can get answers to solve even the most troublesome bugs.
The Windows SDK enables you to decide which
programming model you want to use. Windows
Forms (either COM or managed) and WPF are
available for you to use in your Tablet PC and
mobile PC applications. In addition to the documentation available for each of these, the SDK
loads samples into “C:\Program Files\Microsoft SDKs\Windows\v6.0\Samples\TabletPC”
directory.
COM
The Windows SDK installs the following header
files that pertain to Tablet and Touch Technology.
These are saved by default in “C:\Program Files\
Microsoft SDKs\Windows\v6.0\Include”.
• msinkaut.h contains the core ink capabilities
for COM.
• IACOM.h contains the InkAnalysis functionality for the COM layer.
• RTSCOM.h contains the RealTimeStylus and
StylusPlugin calls for COM.
.NET Framework Windows Forms
The Windows SDK installs all the managed DLLs
for Tablet and Touch Technology, which include
the following namespaces that contain Tablet and
Touch Technology APIs. (Don’t forget using or
imports statements in your applications):
• Microsoft.Ink. This namespace contains APIs
that implement digital ink on the Tablet PC
and was the original managed namespace from
the first release of the Tablet PC API. This has
all of the managed elements for ink, recognition, rendering, and persisting ink. Additionally, starting with Windows Vista, the managed layer of the InkAnalysis API resides in
this namespace. InkAnalysis gives developers
the ability to parse free-form ink into words,
lines, paragraphs, bullets, and other meaningful data.
• Microsoft.Ink.TextInput. New in Windows
Vista, this namespace contains the APIs exposed by Tablet PC Input Panel.
• Microsoft.StylusInput. This namespace contains APIs that handles the stylus packet data
from a digitizer in real time. It has become
known as the RealTimeStylus API because of
the major object in the namespace.
• Microsoft.StylusInput.PluginData. This name
space contains the data passed to RealTimeStylus plug-ins.
Windows Presentation Foundation
Windows Presentation Foundation (WPF) provides
the basis for building applications and high fidelity
experiences in Windows Vista, blending together
application UI, documents, and media content,
while exploiting the full power of your computer.
The functionality extends to the support for Tablet
PC, UMPC, and other forms of input, a more modern imaging and printing pipeline, accessibility and
UI automation infrastructure, data-driven UI and
visualization, as well as the integration points for
weaving the application experience into the Windows shell.
You can also use the following namespaces that are
directly designed to be used with WPF.
• System.Windows.Controls. Use the InkCanvas control, an element in this namespace, to
receive and display ink input. Generally done
through a stylus, though you can also use a
mouse with a digitizer to produce ink strokes.
The created strokes are represented as Stroke
objects, and can be manipulated programmatically and by user input. If you only need to
render ink onto a surface (rather than recognize or parse the ink) the InkPresenter control renders ink on a surface.
• System.Windows.Ink. Provides classes to interact with and manipulate ink on the Windows Presentation Foundation (WPF) platform.
• System.Windows.Input. Contains a lot of
classes that provide access to the stylus, gestures, and other important classes related to
ink and commands.
Getting the Goods
Here is a list of the resources
mentioned in this article,
as well as some additional ones.
Mobile PC Developer Center
The home for articles,
downloads, samples, news,
videos, and more for developers
targeting notebooks, Tablet PCs,
and UMPCs.
You can find the rest of the
resources in this sidebar through
the developer center.
http://msdn.microsoft.com/
mobilepc
Windows Software
Development Kit for Windows
Vista and .NET Framework 3.0
Runtime Components
SDK includes content for
application development
with the APIs in Windows
Vista, including the .NET
Framework 3.0 technologies:
.NET Framework 2.0, Windows
Presentation Foundation,
Windows Communication
Foundation, Windows Workflow
Foundation, and Windows
CardSpace™.
http://www.microsoft.com/
downloads/Search.aspx
39
Microsoft Windows XP
Tablet PC Edition Software
Development Kit 1.7
Enables you to build inkenabled, pen-enabled,
and speech-enabled applications
and supporting hardware for the
Tablet PC.
Microsoft Windows XP
Tablet PC Edition 2005
Recognizer Pack
Includes the handwriting
recognition engines for
Chinese (Traditional), Chinese
(Simplified), U.S. English, U.K.
English, French, German, Italian,
Japanese, Korean, and Spanish.
Update for Microsoft
Windows XP Tablet PC Edition
Development Kit 1.7
Extends version 1.7 of the
Development Kit by adding new
ink analysis API’s and a COM
version of the Stylus Input API’s.
Microsoft Windows Journal
Reader Supplemental
Component for the Windows XP
Tablet PC Edition Development
Kit 1.7
Provides programmatic read
access to files in the Journal
format.
Ultra-Mobile PC (UMPC)
Display Emulator
Enables you to test your
application’s layout and screen
behavior as it would appear on
a UMPC.
40
Figure 1: Strokes can share DrawingAttributes.
• System.Windows.Input.StylusPlugIns. Provides classes that support manipulating data
from a tablet pen in WPF. Essentially, this is
the WPF version of the StylusInput managed
API.
With the various layers available, you can decide
which platform is best suited for your mobile PC
needs. You can use WPF for applications deployed
on computers running Windows Vista, but WPF is
also backward-compatible on computers running
Windows XP. One development hitch is ensuring
that you use the correct version of InkAnalysis
or StylusPlugin APIs, as there are three of each—
COM, managed, and WPF.
You Changed What?
Figure 2: StrokeCollection objects can share Stroke objects.
On the WPF platform, the System.Windows.Ink.
Stroke class owns and manages its own lifetime.
You can collect a group of Stroke objects together
in a StrokeCollection, which provides methods for
common ink data management operations such as
hit testing, erasing, transforming, and serializing
the ink. A Stroke can belong to zero, one, or more
StrokeCollection objects at any give time. Instead
of having a Microsoft.Ink.Ink object, the InkCanvas and InkPresenter contain a System.Windows.
Ink.StrokeCollection.
On the Windows Forms and COM platforms, the
Microsoft.Ink.Ink object constrains the lifetime of
the Microsoft.Ink.Stroke objects, and the stylus
packets belong to the individual strokes. Two or
more strokes can reference the same Microsoft.Ink.
DrawingAttributes object, as shown in Figure 1.
WPF and Windows Forms have different object
models for collecting, storing, manipulating, and
rendering ink. With essentially three platforms
that support digital ink, the Windows Forms and
COM platforms share a similar object model, but
the object model for the WPF platform is newly architected.
In WPF, each System.Windows.Ink.Stroke is a
common language runtime object that exists as long
as something has a reference to it. Each Stroke references a StylusPointCollection and System.Windows.Ink.DrawingAttributes object, which are also
common language runtime objects (Figure 2).
All three platforms ship objects and controls that
enable an application to receive input from a tablet
pen. The Windows Forms and COM platforms ship
with InkPicture, InkEdit, InkOverlay and InkCollector classes. InkPicture and InkEdit are controls
that you can add to an application to collect ink.
You can attach InkOverlay and InkCollector to an
existing window to ink-enable windows and custom
controls. The WPF platform includes the aforementioned InkCanvas and InkPresenter controls.
I’m Working on XP Here
On the Windows Forms and COM platforms, InkCollector, InkOverlay, InkEdit, and InkPicture
each expose a Microsoft.Ink.Ink object. The Ink object contains the data for one or more Microsoft.Ink.
Stroke objects and exposes common methods and
properties to manage and manipulate those strokes.
The Ink object manages the lifetime of the strokes
it contains; the Ink object creates and deletes the
strokes that it owns. Each Stroke has an identifier
that is unique within its parent Ink object.
You can still install the Windows Vista SDK onto
your Windows XP development computer and insert the Tablet and Touch Technology within your
applications. And if you are one of the developers
who are using a Tablet PC as your development environment, you are all set. If your developer computer is not a Tablet PC, you should first download
and install the Windows XP Tablet PC Edition Software Development Kit 1.7 and then download the
Windows Vista SDK.
Some of you are still using Windows XP as your
operating system. If you fall into this category, then
there are some special considerations to take in order to develop and test Tablet PC and UMPC applications fully.
Windows Vista SDK
Remember, too, that I mentioned that the recognizers are incorporated directly into premium versions
of Windows Vista. The same, however, is not true
for computers running Windows XP (except for
computers running Windows XP Tablet PC Edition). In order to enable handwriting recognition in
this case, install the Microsoft Windows XP Tablet
PC Edition 2005 Recognizer Pack, after first installing the Tablet PC SDK. (Although the recognizers
and their functionality are not redistributable, you
are free to use them on your development computer
in order to build, troubleshoot, and test your applications.)
The Recognizer Pack provides ten handwriting-recognition engine languages:
•
•
•
•
•
•
•
•
•
•
English (US)
English (UK)
French (France)
German (Germany)
Italian
Japanese
Korean
Chinese (Simplified)
Chinese (Traditional)
Spanish (International Sort)
Tablet PC Platform SDK
If you cannot or do not wish to install the Windows Vista SDK, you can still create highly robust
Tablet PC and UMPC applications. The most glaring loss of functionality is that you cannot use the
Windows Presentation Foundation for rendering,
nor can you take advantage of the APIs for the
Tablet PC Input Panel that ship with the Windows
Vista SDK.
Having said that, you still have exceptional APIs
with which to work. As I mentioned, the latest version of the Tablet PC Platform APIs is the Tablet
PC SDK 1.7, available through MSDN Downloads.
The Tablet PC SDK 1.7 documentation contains
reference and conceptual topics about core ink
scenarios, the StylusInput APIs (for managed
code), and details about ink-enabling Web pages.
You’ll find many samples and updates to topics
from earlier versions. The SDK also contains the
COM version of ink functionality, the SetInputScope API reference, and the following managed
namespaces:
• Microsoft.Ink
• Microsoft.StylusInput
• Microsoft.StylusInput.PluginData
You can add further functionality by installing
the Update for Microsoft Windows XP Tablet PC
Edition Development Kit 1.7, from MSDN Downloads. This update includes a COM version of the
StylusInput API, as well as the InkAnalysis API
for COM and managed Windows Forms. (Note
that the version of InkAnalysis for Windows Presentation Foundation is only available in the Win-
dows Vista SDK.) Again, to enable handwriting
recognition on a non-Tablet PC running Windows
XP, install the Microsoft Windows XP Tablet PC
Edition 2005 Recognizer Pack after installing the
SDK.
One avenue of functionality that enables crossplatform scenarios is the Windows Journal Reader
Supplemental Component for the Windows XP
Tablet PC Edition Development Kit 1.7. One of
the first applications to support ink was Windows
Journal. For a while, it was the shining example
of how a Tablet PC running an ink-enabled application could provide for a great user experience.
Using the Journal Reader component, you can develop an application that converts Journal notes
to an XML format. From the Journal XML format
you can either open and view Journal notes in your
own application or convert them to a new format.
For more information about the Journal Reader,
see Converting Journal Notes to XML, SVG, and
OneNote in the MSDN Library.
Sharing Between Platforms
Although the platforms have different object models for the ink data, sharing the data between the
platforms is very easy. The code example in Listing
1 saves ink from a Windows Forms application for
use in a WPF application.
For more information about the differences between the object model for ink in WPF and the
.NET Framework, take a look at The Ink Object
Model: Windows Forms and COM versus WPF in
the MSDN Library.
Microsoft UMPC Portal Page
Full resources and information
for and about UMPC.
umpc
Origami Project
The Microsoft UMPC Team
maintains this site with
information, blogs, and more.
http://origamiproject.com
Get Started Writing Gadgets for
Windows SideShow Devices
Focuses specifically on SideShow
hardware devices and how to
write a SideShow gadget to
communicate with these devices
using the managed API.
http://msdn.microsoft.
com/msdnmag/issues/07/01/
SideShow/default.aspx
Windows SideShow Platform
Documentation
Comprehensive guide for creating
Windows SideShow gadgets.
http://msdn2.microsoft.com/
en-us/library/ms744202.aspx
Listing 1: Moving ink from Microsoft.Ink to System.Windows.Ink
using
using
using
using
System;
Microsoft.Ink;
System.Windows.Ink;
System.IO;
namespace WinFormsXAMLInkConversion
{
class InkInterop
{
System.Windows.Ink.StrokeCollection
InkToStrokeCollection
(Microsoft.Ink.Ink inkToSave)
{
byte[] savedInk = inkToSave.Save();
MemoryStream theMemoryStream =
new MemoryStream(savedInk);
System.Windows.Ink.StrokeCollection strokes =
new System.Windows.Ink.StrokeCollection
(theMemoryStream);
return strokes;
}
}
}
41
How Do I Write?
Just because you have the software to create inkenabled applications, it doesn’t mean you can pick
up your favorite pen or stylus and write on your
22-inch flat screen. If you are not lucky enough to
have a computer with a built-in digitizer, there are a
number of ways to get around this. First of all, you
can use the mouse for ink input. Mouse commands
are intercepted by the pen thread and will trigger
ink rendering on appropriate surfaces in an application. Because the sample rate for ink is so much
higher than for mouse input,
however, mouse input tends to
be a little jagged. In addition,
Developers of Tablet PC
it’s pretty hard to write cursive characters with a mouse.
and mobile PC applications have
shown both drawbacks in
the expertise gained from using the I’ve
Figure 3.
API for pen and ink, designing for
flexible display layouts,
and network and power awareness
that will enable them to optimize
applications for this
new class of computer.
If you are fully testing an application that you know will
be deployed on a Tablet PC or
UMPC, the best testing scenario is to use the same hardware you’re targeting. If you
are on a tight budget, though,
all is not lost. With a relatively
modest investment, you can
use an external digitizer that
plugs into your computer. I recommend that you
get one of the models with pressure sensitivity, especially if you plan on incorporating that feature
into an application. Figure 4 shows how input using a digitizer flows much more smoothly than a
mouse.
What about UMPC and the Little
Screen?
Developing for Ultra-Mobile PC is just like developing for Tablet PC. The UMPC is a device-like
computer that is small, mobile, and runs the full
Windows operating system (Windows XP Tablet PC
Edition 2005 or Windows Vista). The UMPC accomplishes what many and varied portable devices
Figure 3: Drawing with a mouse has a slower
sample rate than a digitizer.
42
Figure 4: The digitizer’s rapid sample rate results
in smooth and fluid ink rendering.
do today, but because it runs the full Windows XP
operating system, the UMPC does everything your
desktop computer can.
Because the UMPC runs Windows, all of the familiar
Windows APIs are still available to you when you
design applications for it. In addition, developers of
Tablet PC and mobile PC applications have the expertise gained from using the API for pen and ink,
designing for flexible display layouts, and network
and power awareness that will enable them to optimize applications for this new class of computer.
The typical Ultra-Mobile PC has a slate form factor (ie., no physical keyboard), weighs less than 2.5
pounds, and has a LCD display between five and
seven inches in size and with a resistive touch digitizer. With its compact, ergonomic design, the Ultra-Mobile PC is ideal for use in limited attention
scenarios, such as standing or walking. In these and
other mobile situations, users must be able to interact quickly and simply with their UMPCs by using
such methods as direct touch, stylus, hardware control buttons, soft keyboard, dial keys, and speech.
Because of this, the most important factor when developing UMPC applications is being aware of the
display limitations. Ultra-Mobile PC applications
should present information in a form that users can
easily digest at a glance. Overly dense content is
difficult to read quickly in mobile situations, and a
dense user interface layer requires a great degree of
user attention.
To help with this, download the Ultra-Mobile PC
(UMPC) Display Emulator from MSDN Downloads. The UMPC Display Emulator enables you to
test your application within the screen confines of
the UMPC from your higher resolution desktop or
mobile PC. Launch the emulator, use the Windows
UI to run your application, and see how your application works in the smaller format UI and screen
resolutions.
Windows SideShow
Windows SideShow technology enables laptop
manufacturers to include a secondary or auxiliary
display in laptop designs, many of which were unveiled this year at the 2007 International CES. Users that have laptops that support Windows SideShow can easily view critical information, whether
the laptop is on, off, or in sleep mode. The convenience provided by these auxiliary displays will
also save time and battery life by allowing users to
quickly view meeting schedules, phone numbers,
addresses, and recent e-mail messages without
having to start up the laptop.
The Windows Vista SideShow Platform also enables hardware manufacturers to build auxiliary
displays in a wide range of peripheral devices such
as keyboards, LCD display casings, remote controls,
and cell phones. These devices can then display information received from a Windows Vista-based
Figure 5: Possible Windows SideShow
implementations.
PC, providing even more convenience to everyday
computing. Figure 5 shows some possible examples
of Windows SideShow-compatible devices.
To take advantage of Windows SideShow, your
application must register for the service. The Windows Vista Software Development Kit includes a
Windows SideShow-compatible device simulator
that developers can use to test their gadgets without physical hardware. The SDK also includes
sample gadgets that work with the simulator and
illustrate how to build a simple gadget for Windows
SideShow. Reference topics are available both for
COM and managed versions of Windows SideShow. In fact, the January issue of MSDN Magazine has a great article by Jeff Richter, “Get Started
Writing Gadgets for Windows SideShow Devices,”
that details the managed API. You should also review Neil Roodyn’s article, “SideShow Development in Depth” in this issue of CoDe Focus.
User Experience Guidelines for
Developers
In addition to suggestions like those for UMPC, the
“Mobile PC User Experience Guidelines for Developers” (in the MSDN Library) offers a wide range
The Windows Vista
SideShow platform will also enable
hardware manufacturers to build
auxiliary displays in
a wide range of peripheral devices
such as keyboards, LCD display
casings, remote controls,
and cell phones.
of findings pertinent for developing applications
for portable computers. These guidelines describe
how you can optimize the computing experience
for your mobile PC users by following established
user interface design principles in your application,
whether or not the application is designed specifically for mobile computing scenarios. These guidelines include the following topics:
•
•
•
•
•
•
•
•
•
Power and Device Awareness
Responding to Changing Network Conditions
Adapting to a Flexible Display Environment
Supporting Pen and Touch Input
Integrating Ink
Text Input Using the Pen
Command Input on the Tablet PC
Ultra-Mobile PC
Windows SideShow Gadgets
Finally, the Mobile PC Developer Center (http:
msdn.microsoft.com/mobilepc) contains new and
updated content, samples, hands-on labs, videos,
and other resources to ensure that your development environment is up to date and that you have
all the resources and knowledge you need to create
world-class mobile PC applications. All you need
now is the will to develop and the time to write
code. We all know how hard it is to get those. But
when more than half the world’s computers are
mobile computers, it’s time to make them a priority
(see “Will the Notebook Market Hit a Tipping Point
in ‘07?” eWeek.com: http://www.eweek.com/article2/0,1895,2074603,00.asp).
Just because
you have
the software to
create ink-enabled
applications,
it doesn’t mean
you can pick up
your favorite pen or
stylus and write on
your 22-inch flat
screen.
Mobile PC User Experience
Guidelines for Developers
Describes how you can optimize
the computing experience for
your mobile PC users by following
established user interface design
principles in your application.
Eliot Graff
43
Adapting Existing Applications
to Work on UMPCs
Dr. Neil Roodyn
[email protected]
Dr. Neil is an independent
itinerant consultant, trainer,
and author. He travels the
world working with software
companies. Dr. Neil loves
Australia, where he spends the
summer enjoying the Sydney
lifestyle and helping software
development teams get more
productive. Dr. Neil spends his
other summer each year flying
between Europe and the USA
working with software teams and
writing about his experiences.
Neil brings his business and
technical skills to the companies
he works with to ensure he has
happy customers.
The Ultra-Mobile PC (UMPC) presents a new opportunity
for existing applications to extend their potential audience.
Microsoft® Windows® desktop applications can mobilize onto the UMPC platform,
providing users with desktop functionality while on the move. Windows Mobile™
device applications can take advantage of the larger screen size and storage space
of the UMPC to extend the application capabilities.
W
hether you are a Windows Mobile device
plication. The goal in both scenarios is to maintain
developer or a desktop application devela single code base that creates an application that
oper, the UMPC provides exciting new opcontinues to run on the original target platform but
portunities for your applications to reach users who
also provides a great user experience for a user runmight not have previously enjoyed interacting with
ning on an Ultra-Mobile PC.
your software. In order to provide the best user experience on
You can download all the code
Fast Facts
the UMPC, your application is
for this article from http://Roodyn.
Using the .NET Framework,
likely to require some user incom/WorkOnUMPC.ashx
terface changes. If you are lookit is now possible to build
ing to extend the reach of your
a single assembly that can
Windows Mobile application,
Adapting a PC
run on all of your machines,
there may also be some funcApplication to the UMPC
from a Windows Mobile
tionality changes required.
This article will walk through
two scenarios: changing an
existing desktop Windows application to work well on the
UMPC, and then repeating
the process but this time with
a Windows Mobile device ap-
Smartphone through to
a server, with no need to
recompile. As the UMPC is a
full PC in a small form factor,
you can extend the reach
of your application to a new
target audience.
A UMPC is a full PC, running
the full version of the Windows
operating system. This implies
that a PC application should
“just run” on a UMPC and it
will. However, there is a slight
hitch; most PC applications are
built with a view that the user
will be driving the application with a mouse, the
screen size will be at least 1024 x 768 pixels, a keyboard is attached to the computer, and user’s eyes
will be approximately between 20 and 30 inches
from the screen.
A user is likely to drive a UMPC application with
either their finger or a stylus. The default resolution
for the UMPCs available today is 800 x 480 pixels.
Many of today’s UMPCs will not have a keyboard
attached. Microsoft designed the UMPC to be held
in the hand and the user’s face is likely to be approximately 10 to 15 inches from the screen. With
these differences in mind, you can create a UMPC
version of an application.
To resolve the lack
of hover support, you need to think
about what other visual clues you
can provide to the user.
Figure 1: Windows application running on a UMPC.
44
Adapting Existing Applications to Work on UMPCs
The sample PC application converted in this article is
a simple picture viewer that allows the user to build
collections of pictures to browse—think of the collections as playlists for pictures. Figure 1 shows the
application running on a UMPC before it has been
converted. You can see that the window has been
squashed to try and fit it within the screen size and the
right edge of the window is off the screen. The toolbar
buttons are small; unless you have delicate little fingers these would cause an issue for most users.
If you run this application on a PC with a larger
screen and a mouse you will notice that the default
size for the main windows is a quite large 1000 x 700.
You will also notice that each toolbar button shows a
tooltip when the mouse pointer hovers over the button. These tooltips provide the user with information
as to the function of each button. With a UMPC being
operated by finger there is no concept of hover. Unlike an active digitizer Tablet PC, the UMPC touch
digitizer provides no means to hover with a stylus or
your finger. Tooltips, such as those used on the toolbar
buttons, are useless for a UMPC application.
The TreeView and Menu controls will also likely present some issues for a UMPC user using their finger to
interact with the screen. You can address the size of
the buttons, TreeView control nodes, and the menu by
simply increasing their respective sizes and font sizes.
Microsoft provides a document on MSDN® called the
“Mobile PC User Experience Guidelines for Developers” (http://msdn2.microsoft.com/en-gb/library/ms695565.
aspx), which addresses many issues such as control
sizing.
To resolve the lack of hover support you need to think
about what other visual clues you can provide to the
user. In this case, changing the toolbar button style
from Image to ImageAndText provides informative
text on the buttons.
You can resolve the issue with sizing the form by ensuring the form is never bigger than the working area
of the screen. You should then position the form in
the center of the screen. The code in Listing 1 provides an example of a simple approach you should
consider. In order to support features, such as changes
in screen orientation, you will need to get a little bit
smarter. I will explain this further in the next section
as Windows Mobile applications need to deal with
orientation changes as well.
You can see the final result of these changes in Figure 2. The controls are far more finger friendly. As
the UMPC is a full PC there will be no issues with
functionality needing to be changed; it is simply a case
of ensuring the user interface provides a better experience for your users.
Figure 2: Windows application after the user interface changes.
Figure 4: Windows Mobile application running in landscape.
coding, rather than all this mucking around with
the user interface. Although, as you will discover,
it is not as hard as you might fear.
Listing 1: Resize the form
private void ResizeForm()
{
Rectangle screenRect = Screen.GetWorkingArea(this);
if (this.Width > screenRect.Width)
{
this.Width = screenRect.Width;
}
if (this.Height > screenRect.Height)
{
this.Height = screenRect.Height;
}
Point location = new Point(
(screenRect.Width / 2) - (this.Width / 2),
(screenRect.Height / 2) - (this.Height / 2));
this.Location = location;
Adapting a Pocket PC Windows Mobile
Application to the UMPC
Running a Windows Mobile application on the
UMPC requires a little more care and some actual
Figure 3: Windows Mobile
application in portrait.
}
45
Carmen Lui
SDET Lead,
Tablet Experience Team
Text input is getting even
better in Windows Vista. The
accurate and fast handwriting
recognition of Tablet PC Input
Panel unleashes the productivity
of a Tablet PC. Using a pen and
ink on the screen is such an
enjoyable and fun experience.
My favorite feature of Input Panel
is AutoComplete. The automatic
URL matching in the Internet
Explorer address bar makes
entering Web site addresses
quick and easy.
The example application in this case does something
similar to the PC application in the last example, except it is designed to be run on a Pocket PC running
Windows Mobile 5.0. Figure 3 shows the application
running in portrait. As previously mentioned, many
Windows Mobile devices provide the user an option to
change the screen orientation, hence this application
already copes with running in landscape as shown in
Figure 4. When switching from portrait to landscape
the controls change position to provide a better user
experience—notice how the TreeView shifts from being under the image to the left of the image.
This Pocket PC application has two other interesting features. The first is the RenameCollection form,
shown in Figure 5. The RenameCollection form uses
the InputPanel that is specific to the Windows Mobile platform. The equivalent on the UMPC would be
Tablet PC Input Panel.
The other feature of interest is the SelectPictureDialog that you use to allow the user to add a photo to a
photo collection, shown in Figure 6. This is a dialog
box specific to Windows Mobile 5.0 and the nearest
equivalent on a UMPC would be the common dialog
control, OpenFileDialog.
Before resolving these issues specific to Windows
Mobile, you need to be able to run the .NET Compact Framework assembly on a machine hosting
the full .NET Framework. Fortunately for Windows
Mobile developers, with the introduction of the
.NET Compact Framework 2.0, Microsoft added a
new keyword into the MSIL that the language compilers generate: retargetable. If you use ILDASM to
view the intermediate language of an assembly built
using the .NET Compact Framework, you will see
this keyword at the top of the Manifest:
.assemblyextern retargetable System.Windows.Forms
{
.publickeytoken = (96 9D B8 05 3D 33 22 AC )
.ver 2:0:0:0
}
Figure 5: RenameCollection form.
46
This retargetable keyword is going to become your
best friend if you want to have your Windows Mobile applications running on a UMPC—or
any other PC for that
matter. The retargetable keyword informs
the CLR that this
assembly can potentially run on another
compatible
version
of the CLR and not
only the version it is
primarily
targeting.
To determine the version of the CLR the
assembly is primarily targeting, use the
combination of the
publickeytoken and
Figure 6: SelectPictureDialog.
the ver values.
Do you remember what your mentor taught you
about never hard coding file paths and always getting
environment information programmatically? Well
it was all true. If you build a Windows Mobile application using the .NET Compact Framework 2.0 and
you stick to these good programming practices, you
will find you can have your application running on
a full PC device (such as a UMPC) with ease.
Remember what your
mentor taught you about never
hard coding file paths and always
getting environment
information programmatically?
Well it was all true.
If you have downloaded the source code for this
article, you can try running the Pocket PC application on your PC. As long as you have the .NET
Framework 2.0 (or greater) installed, the application will run. As shown in Figure 7, the size of the
application is maintained at Pocket PC size. You
need to change this in order to provide a better user
experience on the UMPC.
There are also issues with the RenameCollection
form and SelectPictureDialog. Selecting either of the
menu items (Rename or Add Photo) that create and
run these dialog boxes will cause an unhandled exception to be thrown and the application will crash.
The InputPanel being displayed in the RenameCollection form is being caught by the CLR JIT compiler as an invalid class. The offending code is in the
GotFocus event handler for the TextBox:
private void renameTextBox_GotFocus(object sender,
EventArgs e)
{
Microsoft.WindowsCE.Forms.InputPanel ip =
new Microsoft.WindowsCE.Forms.InputPanel();
ip.Enabled = true;
}
You may think you could solve this by only calling the code when running on a Windows Mobile
(WinCE) device. Something like this:
EventArgs e)
{
if (System.Environment.OSVersion.Platform ==
PlatformID.WinCE)
{
new
Microsoft.WindowsCE.Forms.InputPanel();
ip.Enabled = true;
}
}
However, this will not work as the CLR JIT compiler attempts to compile an entire method at a
time, not just the code path being taken in this code
run (this makes sense when you think about it). The
next step is to extract the contents of the if scope
into another method:
This would appear to be the solution as the CLR JIT
compiler will only compile the ShowInputPanel
method if the method is called and that would mean
the platform on which the code is running is Windows Mobile. However there is one more obstacle
to overcome. Can you guess what it is?
EventArgs e)
{
PlatformID.WinCE)
{
ShowInputPanel();
}
}
Think about how the code is compiled. Before the
JIT compiler gets to the code it is compiled from
C# (or Visual Basic®) into MSIL. During that process the code might be optimized. This optimization process might include inlining some methods,
especially short methods that are not called from
anywhere else.
private static void ShowInputPanel()
{
ip.Enabled = true;
}
To prevent the code from being inlined, add an
attribute to the method that is a compiler directive:
Figure 7: Pocket PC application
running on PC.
MethodImplAttribute(MethodImplOptions.NoInlining)
Using this technique for both the GotFocus and LostFocus event handlers in the RenameCollection form,
you can see the resulting code in Listing 2.
Listing 2: Showing the InputPanel in Windows Mobile only
private void renameTextBox_GotFocus(object sender, EventArgs e)
{
if (System.Environment.OSVersion.Platform == PlatformID.WinCE)
{
ShowInputPanel();
}
}
private void renameTextBox_LostFocus(object sender, EventArgs e)
{
if (System.Environment.OSVersion.Platform == PlatformID.WinCE)
{
HideInputPanel();
}
}
[MethodImplAttribute(MethodImplOptions.NoInlining)]
private static void ShowInputPanel()
{
ip.Enabled = true;
}
private static void HideInputPanel()
{
ip.Enabled = false;
}
Listing 3: Using the SelectPictureDialog in Windows Mobile only
private void addPhotoMenuItem_Click(object sender, EventArgs e)
{
TreeNode selected = collectionTreeView.SelectedNode;
if (selected != null)
{
PlatformID.WinCE)
{
WindowsMobileAddPhoto(selected);
}
else
{
FullWindowsAddPhoto(selected);
}
}
}
}
private void WindowsMobileAddPhoto(TreeNode selected)
{
Microsoft.WindowsMobile.Forms.SelectPictureDialog dlg =
new Microsoft.WindowsMobile.Forms.SelectPictureDialog();
private void FullWindowsAddPhoto(TreeNode selected)
{
OpenFileDialog dlg = new OpenFileDialog();
dlg.InitialDirectory =
Environment.GetFolderPath(Environment.SpecialFolder.Personal);
dlg.Filter =
"Image Files(*.BMP;*.JPG;*.GIF)|*.BMP;*.JPG;*.GIF|All
files (*.*)|*.*";
DialogResult result = dlg.ShowDialog();
if (result == DialogResult.OK)
{
AddPhotoToCollection(dlg.FileName, selected.Text);
}
}
dlg.InitialDirectory = Environment.GetFolderPath(
Environment.SpecialFolder.Personal);
dlg.Filter = "Image
Files(*.BMP;*.JPG;*.GIF)|*.BMP;*.JPG;*.GIF|All
files (*.*)|*.*";
DialogResult result = dlg.ShowDialog();
if (result == DialogResult.OK)
{
AddPhotoToCollection(dlg.FileName, selected.Text);
}
47
With this knowledge you can easily solve the issue
with the SelectPictureDialog. The code in Listing 3
shows how you can use the SelectPictureDialog on a
Windows Mobile device and the OpenFileDialog on
a PC. Examine the code in Listing 3 (and if you know
a little something about how I work) you may wonder
why it appears that I’ve duplicated the code in the
FullWindowsAddPhoto and the WindowsMobileAddPhoto methods. The only difference is in the creation
of the dlg object. Unfortunately FileOpenDialog and
the SelectPictureDialog do not share the same base
class; therefore, refactoring this code to reuse the code
becomes less trivial.
Now it is time to turn attention to the user interface
aspects of the application. A ScaleForm method handles the orientation and scaling issues (Listing 4). This
method determines the orientation and then lays out
the control accordingly. This works well on Windows
Mobile devices that all have relatively small screens. It
also deals well with changes in the DPI.
Now I’ll address three user interface issues: when the
application runs on the UMPC the default window
size is still the small Windows Mobile size, the but-
Figure 8: Pocket PC application running on a UMPC.
Listing 4: Handling orientation and scaling
private void ScaleForm()
{
if (ClientSize.Height > ClientSize.Width)
{
//portrait
//put collections list below photos
collectionTreeView.Location =
new Point(0, ClientSize.Height / 4*3);
collectionTreeView.Width = ClientSize.Width;
collectionTreeView.Height = ClientSize.Height / 4;
}
else //landscape or square
{
//put collection list on left of photos
collectionTreeView.Location = new Point(0, 0);
collectionTreeView.Width = ClientSize.Width / 4;
collectionTreeView.Height = ClientSize.Height;
photoPictureBox.Location =
new Point(collectionTreeView.Width, 0);
photoPictureBox.Width = ClientSize.Width collectionTreeView.Width;
photoPictureBox.Height = ClientSize.Height;
prevPictureBox.Location = new Point(photoPictureBox.Left,
photoPictureBox.Height - prevPictureBox.Height);
nextPictureBox.Location = new Point(photoPictureBox.Right nextPictureBox.Width, photoPictureBox.Height –
nextPictureBox.Height);
photoPictureBox.Location = new Point(0, 0);
photoPictureBox.Width = ClientSize.Width;
photoPictureBox.Height =
ClientSize.Height - collectionTreeView.Height;
prevPictureBox.Location = new Point(photoPictureBox.Left,
photoPictureBox.Height - prevPictureBox.Height);
nextPictureBox.Location = new Point(photoPictureBox.Right –
nextPictureBox.Width, photoPictureBox.Height nextPictureBox.Height);
}
}
Figure 5: Sizing the controls when executed on a PC
private void PocketPhotos_Load(object sender, EventArgs e)
{
if (System.Environment.OSVersion.Platform != PlatformID.WinCE)
{
Rectangle screenRect = Screen.PrimaryScreen.WorkingArea;
this.Width = screenRect.Width;
this.Height = screenRect.Height;
this.Location = new Point(0, 0);
prevPictureBox.Width = prevPictureBox.Width * 2;
prevPictureBox.Height = prevPictureBox.Height * 2;
48
nextPictureBox.Width = nextPictureBox.Width * 2;
nextPictureBox.Height = nextPictureBox.Height * 2;
collectionTreeView.Font = new
Font(FontFamily.GenericSansSerif, 14,
FontStyle.Regular);
}
ScaleForm();
}
Figure 9: UMPC Display Emulator.
tons seem too small, and the TreeView items could be
more finger friendly.
The ScaleForm method is called when you first load
the form and also each time the device resizes it, this
way it can deal with a device switching from portrait
to landscape. You only need to load the code for increasing the size of the controls to be finger friendly
for a UMPC once, so add it in the form’s Load event
handler, as shown in Listing 5.
If you now execute the assembly on a UMPC you’ll
see that it provides a far better user experience as
well as a fully functional application (Figure 8). The
fantastic thing about this code is that it will run on
either a Pocket PC with the .NET Compact Framework 2.0 installed or a full PC (such as a UMPC)
with the .NET Framework 2.0 (or greater) installed.
You do not need to recompile the code to move between platforms.
Testing Your UMPC Application
Nothing beats actually running the application on a
real UMPC device. This is a good reason to always
have an up-to-date installer for your application. A
simple installer will make the deployment simpler and
testing the application on a real device a smoother
experience.
During development, running the application on a
UMPC while testing the user interface features can
prove tedious. Developing the code on the UMPC
(while possible) will become even more stretching for
most developers. Let me offer a few tricks to use.
First consider doing the development in a virtual PC
(VPC) environment. This has several advantages, and
in particular for UMPC development you can change
the VPC screen size to match the UMPC screen resolution. If you have enough memory on the development
machine, it is worth creating a second VPC image that
is set up and running with the same software as on
your UMPC and the screen resolution to match.
The UMPC Display Emulator from Microsoft offers
another great way to validate your application. It adjusts the resolution of your screen to 800 x 480 pixels
and fits it inside an image of a UMPC; Figure 9 shows
the UMPC Display Emulator. You can download the
emulator from: http://www.microsoft.com/downloads/details.aspx?FamilyID=9C01C822-E6D2-4240-9FB3-B00E8FF4F
BBF&displaylang=en or find it through the Mobile PC
Developer Center (http://msdn.microsoft.com/mobilepc)
Final Thoughts
The UMPC hardware platform offers new opportunities for everyone. Windows developers will find their
applications run on UMPCs, and with a few user interface changes, these PC applications can provide
a great mobile PC experience. Developers who target Windows Mobile devices can extend the reach
of their Pocket PC applications to the new Mobile
PC hardware options with very little disruption to
their existing code and whilst still maintaining support for their Windows Mobile customers from the
same code base.
Neil Roodyn
49
Introduction to Windows
Touch Technology
in Windows Vista
Hilton Locke
Hilton Locke is a Software
Development Engineer in Test
at Microsoft Corp. He has been
working on the Tablet PC team
since before version 1.
A hardware junkie at heart, he
lives for that just-unwrappeda-new-component feeling.
His current assignment on the
Mobile PC team allows him to
experience that feeling quite
often.
Hilton lives in Sammamish,
WA with his wife and two youngbut-promising engineers.
You can find Hilton’s blog at
http://blogs.msdn.com/hiltonl
I think we’re making progress toward the goal of bending
computers to conform to human factors rather than bending my
human factors to conform to the way computers accept input.
Ultimately, our brains will be plugged directly into “the grid,” but until then, users still
have to learn how to type, control a mouse, and press buttons in certain sequences. The
arrival of the Windows Vista™ operating system with Microsoft® Windows® Tablet and
Touch Technology is another step toward really natural computing.
T
he Mobile PC team’s main vision for Windows
Vista development was to enable people to be
more productive, in more places, with devices
that are more personal.
In More Places…
Implementing touch input in your applications
enables new scenarios and complements existing
mobile scenarios. Imagine reading e-books on the
If you’ve already had the pleasure of using a Tabsubway. Users don’t want fumble around trying to
let PC, you know that there’s
find their stylus in order to turn
a profound difference between
the pages. With touch they can
Fast Facts
the traditional input methods
simply flick a finger to turn the
Support for touch devices is
of keyboard and mouse versus
pages.
new to Windows Vista.
a stylus. Get ready for the next
Touch is a whole new input
step in the progression, WinHave you ever seen somebody
dows Touch Technology.
actually using their laptop
mode that is the most natural
while they were in line at the
and most intuitive
movie theater? No, of course
method yet for interacting
More Productive…
not. But with touch input on an
with Microsoft Windows.
Ultra-Mobile PC or Tablet PC,
Tablet PCs and Ultra-Mobile
it becomes a bit more realistic
PCs (UMPCs) are most useful by improving existing
to imagine buying your movie tickets online while
usage scenarios. Corridor warriors, healthcare proyou’re waiting for your check at dinner.
fessionals, and other on-the-go knowledge workers
are all examples of users with existing scenarios that
Windows Tablet and Touch Technology embodies
are improved by the introduction of mobile PCs.
the goals of Windows Vista by being usable, comfortable, and productive in any place at any time.
But what about categories where it’s difficult or inconvenient to use a stylus?
…Devices that Are More Personal
On a factory floor, workers may have protective
gloves they need to wear while interacting with
their PC. In the field, workers
may already have tools in their
hands, and juggling a stylus
along with their tools just isn’t
complements
practical.
Touch
other input modes and extends
mobile scenarios.
Enter Windows Tablet and
Touch Technology, the natural next step in the evolution
of human-computer interaction. With support for both of the currently leading
technologies behind touch input, Windows Vista is
well-positioned to make users more productive, no
matter where they happen to be.
52
Introduction to Windows Touch Technology in Windows Vista
As device form factors become more like personal
devices (i.e. Smartphones, Pocket PCs, Ultra-Mobile PCs), the traditional methods of interacting
with such devices become less convenient. Does
anybody pull out a full-size keyboard when they use
their Pocket PC or mobile phone? With the arrival
of touch input, it’s easier to successfully tap out an
instant message by pressing on-screen buttons predefined for “LOL” and “On my way” rather than the
current methods which involve a lot of typing.
Touch input creates a more personal connection
between users and their computing devices. When
you think of how you interact with touch-enabled
devices today, those that probably come to mind
Make PCs more useful, more personal
< 4lbs
Desktop
Laptop
Figure 1: Today’s device spectrum.
6” - 7”
UPPC
<
_ 5”
UMPC
Personal devices
Pointer
Left mouse button
Right mouse button
first are ATMs, information kiosks, and PDAs; they
represent activities that are private and personalized to you.
Now that Windows Tablet and Touch Technology
is available in Windows Vista, you can begin to
imagine the personal relationship that will develop
between a user and their devices. Imagine booksized devices aimed at that distinctly twenty-firstcentury version of a diary—the blog? Or maybe you
can remember drawing on your textbook covers
when you were attending school? You probably
felt a powerful sense of “this belongs to me” that
came from being in constant physical contact with
them. Touch-enabled PCs instill a similar sense of
ownership.
In time, touch-enabled PCs could become the modern equivalent of a wallet/pocketbook in that they
both can carry the scraps of data necessary to modern life and everybody needs their own instance.
Hardware manufactures have introduced a variety
of new mobile PC form factors (Figure 1) such as
the Ultra-Mobile PC (7” diagonal), small-form-factor Tablet PCs (8” diagonal, higher resolution) and
others. These kinds of personal devices demand
a better interaction model than keyboards and
mice.
User Experience Ethnographer,
Mobile & Tailored Platforms
Division
Drag area
Figure 2: Anatomy of the touch pointer.
Touch pointer is a small visual aid shaped like a
two-button mouse (Figure 2). It’s composed of a
left-click button, right-click button, and a drag area.
The system cursor stands ready a short diagonal distance away from the touch pointer, so that while
dragging the touch pointer, you can see the cursor
while you carefully target the small items on your
desktop. By small items, I mean buttons, resize handles, minimize/maximize/close boxes, checkboxes,
etc.
Touch enables new
scenarios for mobile devices and
opens new locations where they
can be used.
Touch Pointer
Note that the cursor is not limited to one standard
position away from the touch pointer. Dragging the
touch pointer quickly toward a corner, or side of
the display, will flip it around to another position,
so that you’re never prevented from accessing even
the deepest parts of the screen.
Your fingertip has very different properties from a
Tablet PC stylus. To start with, your finger is relatively large compared to a stylus’ tip. How do you
know that you’re really pointing at what you think
you’re pointing at? Your fingertip may obscure the
area you’re pointing at!
At its simplest, think of the touch pointer as a sidecar attached to the system cursor. No, it’s not a
useless appendage to be made fun of in numerous
bad movies. It is a visual aid to help users see what
they’re targeting and have confidence that they’ve
successfully targeted objects on-screen.
These observations mean that users need a way to
know what they are pointing at. They need confidence that they are actually targeting what they
think they are targeting. The touch pointer, along
with a new set of touch-friendly design guidelines,
offers a better solution.
Listing 1 shows the C# pseudo-code snippet that
allows you to enable or disable the touch pointer
for your application.
Nelle Steele
You can add a button to your Windows taskbar
that enables you to turn off the touch pointer
How do real people in the real
world use UMPCs, Tablet PCs,
and other mobile PCs? What
do they love about them? What
drives them crazy? The yearlong Digital Lifestyle 20/20
project is an attempt to answer
these questions. 32 households
(including students, families, and
young professionals in Seattle
and Tempe) have been given a
wide variety of mobile PC form
factors. Through a series of
site visits and interviews, we’re
observing usage patterns, needs,
and barriers to use in the context
of their busy and complex lives.
Herry Sutanto
Software Development
Engineer,
Tablet Natural UI Team
With the right set of gestures,
your mobile applications
can be used in a much more
effective way.
53
Listing 1: Enable/disable the touch pointer (C#)
const int WM_TABLET_QUERY_SYSTEM_GESTURE_STATUS = 731;
}
const uint SYSTEM_GESTURE_STATUS_TOUCHUI_FORCEON = 0x00000100;
const uint SYSTEM_GESTURE_STATUS_TOUCHUI_FORCEOFF = 0x00000200;
if (...)
{
result |= SYSTEM_GESTURE_STATUS_TOUCHUI_FORCEOFF;
}
protected override void WndProc(ref Message msg)
{
switch (msg.Msg)
{
case WM_TABLET_QUERY_SYSTEM_GESTURE_STATUS:
{
uint result = 0;
if (...)
{
result |= SYSTEM_GESTURE_STATUS_TOUCHUI_FORCEON;
Touch Input
Supporting Cast
Windows Vista offers two
important enhancements to
support touch: panning hand
and recognizers that learn your
individual writing style.
Internet Explorer 7 improves
document navigation with a new
feature called panning hand.
Instead of using the scrollbars to
page through a document, you
can simply grab the page with
touch and move it directly. You
can toggle panning hand on or
off by tapping the button with
the hand icon on the IE toolbar
(Figure 4).
Also new to Windows Vista is
the ability of the handwriting
recognizer to learn your
individual handwriting style.
Invoke the personalization tool
from the Tablet PC Input Panel
by tapping Tools and then
tap Personalize Handwriting
Recognition (Figure 5). If you
don’t want to spend the time to
teach the recognizer through
the wizard, you can turn on
Automatic Learning. From the
Control Panel, choose Tablet PC
Settings, choose Handwriting
Recognition, and then choose
the Use automatic learning
radio button.
With this new capability, you can
teach your Tablet PC that you
write your sevens and zeds with
a slash in them, or perhaps you
slash your zeroes to distinguish
them from oh’s.
54
msg.Result = (IntPtr)result;
}
break;
default:
base.WndProc(ref msg);
break;
}
}
when you want to work directly with the cursor
with touch. This button only works with touch input, you can’t toggle it using your stylus, there’s no
need to worry about accidentally disabling your
touch pointer.
ments has never been easier, because flicking up or
down moves the page you’re viewing up or down
respectively. I love using flicks for reading long Web
pages that extend beyond the bottom of the browser
window.
Users can also toggle the button on or off by rightclicking on the taskbar and clicking Properties.
Navigate down through Taskbar and Start menu to
Properties > Toolbars > Touch Pointer.
You can configure flicks too (Figure 3). You can use
the basic four pre-assigned navigational flicks, or
you can assign your own custom actions to any of
the eight combined navigational and editing flicks
(up, up-right, right, down-right, down, down-left,
left, up-left). If you use cut-copy-paste a lot, you can
enable pre-assigned editing flicks. Virtually anything you can do with a keystroke, you can assign
to a flick.
Navigation Gets Easier
Imagine you’re sitting at home with your Tablet
PC in hand, on your sofa, and you’re yearning to
read that new e-book you’ve just downloaded. After the book loads, you notice that the e-reader
you’re using only recognizes the arrow keys for
navigation. Bummer! Some Tablet PCs don’t even
have keyboards or, if they do, you don’t have access to them without flipping up the screen. What
a pain!
Enter flicks. Flicks are a specialized kind of gesture
that only works with stylus or touch input. All you
have to do to use them is perform an action—you
flick. Flicking your pen to the left generates a left
or back action. Flicking your pen to the right generates a right or forward action. Better yet, flicks
work the same whether you generate them with the
pen or with your finger. Reading long Word docu-
Touch and the Windows
Presentation Foundation
Windows Presentation Foundation (WPF) provides
the foundation for building applications and highfidelity experiences in Windows Vista, blending together application UI, documents, and media content. The functionality of WPF extends to support
Tablet PCs, and by extension, touch input.
If you write WPF (C# or XAML) applications, all
HID-class devices are treated equally as if they were
styluses. This means that they benefit from an established programming model, ink smoothing, and
a host of other benefits. There is no need to modify
any of your WPF apps to support touch, you get it
for free!
Design Considerations for Touch
Figure 3: Control Panel settings for flicks.
Figure 4: Panning hand
in Internet Explorer.
I mentioned earlier, when
introducing
the
touch
pointer, that it was designed
to help bridge the gap between the coarse targeting
of a fingertip and the busy
space most users call their
desktop. As desktop resolu-
tions get higher and higher, and desktop screens
get bigger and bigger, the visual elements drawn on
those screens get closer and closer together. How
do you expect an average user to double-click with
their finger on a single control that contains 20
items?
With the introduction of touch input, developers
must take a small step back and think about how
they guide users to interact with their applications.
A few small changes may be all you need to make
your apps touch-friendlier.
• Provide interface elements that are big enough
for average people to target using touch.
MSDN® says this size is about nine square millimeters (9mm^2) (Figure 6).
• Avoid crowding visual objects too close together. Put at least five pixels of whitespace
between controls, or if the objects have to be
close together, increase the hotspot size (Figure 7).
• Increase the size of the area around the control that users can touch (Figure 8). By making
the hotspot larger than the visible control, you
enable users to select it with less precision.
• Don’t penalize users who mistakenly touch
the “wrong” item of a set. Make it easy for a
user to recover from such accidental touches.
• Use common controls when possible so that
your apps can take advantage of enhancements for pen and touch users for free.
• Use full-screen forms where possible. This
reduces the likelihood of users accidentally
activating a window below your application.
However, you should avoid placing buttons on
the edges of the screen. Users are very likely to
touch the edge of the screen accidentally.
• Minimize the need for users to perform text
entry in touch applications. Make it possible
for users to choose items from a list, or provide
buttons for Yes/No responses.
• Consider whether your users will use the touch
pointer to navigate an un-optimized interface
or implement a touch-friendly user interface
that can be easily acted upon without the
touch pointer.
The touch pointer makes navigating and controlling a touch-enabled device very easy, but it’s still a
good practice to design for human factors. My eyes
are not getting any younger, so I’d prefer you avoid
filling up every pixel of a 1024x768 or 1280x1024
screen. I don’t recommend that you “dumb down”
your forms. Just plan for the ergonomic realities of
touch input.
Figure 5: Handwriting Personalization wizard.
The relevant C++ code looks like this:
if (hr != E_NOINTERFACE)
{
TabletDeviceKind kind;
Chk(ifTablet2->get_DeviceKind(&kind));
if (kind == TDK_Touch || kind == TDK_Mouse)
Figure 6: Ensure that interface controls are at least 9 mm^2.
Figure 7: Allow at least five
pixels of space between adjoining
controls.
Developing Applications that
Understand Touch
Programmatically, to determine what kind of digitizers are available, you’d use the IInkTablet interface
to enumerate how many digitizers are available, and
what capabilities they have.
Figure 8: Enlarge the hotspot around small controls.
55
fTouch = true;
}
else
{
Chk(ifTablet->get_Name(&bstrName));
fTouch = !wcscmp(bstrName,
L"\\\\.\\DISPLAY1");
}
The interesting part that you need to know is how
to use the Tablet_Device_Kind enumeration.
In this way, it’s simple for developers to query,
on pen down, for what kind of input it is and
perform custom rendering on the input appropriately. For example, if a pen is detected, you could
lay down medium-point ballpoint ink. If you find
the input is touch-generated, you could query the
area of the contact, set the cursor size to be the
area of the contact, and apply a smudge effect, or
finger paint. With pressure information, maybe
you could have a faux 3-D “ink” that indicates
where you drew. The possibilities are yours to
discover.
Hardware Considerations
Digitizers: Resistive or Capacitive?
There are two basic technologies underlying touch
digitizers today: active and passive. Passive, or resistive technology, is the sort of digitizer you’ve seen
used in PDAs or touch-screen kiosk displays. In this
kind of digitizer, all the electronic components are
contained in the digitizer itself. The stylus typically
contains no electronics and can be pretty much any
hard object such as a solid plastic stylus or your fingernail.
sistive digitizers. Either you’ve pushed hard enough
or you haven’t. But for capacitive touch, the fact
that there is an electrical current passed into the
screen can be measured to determine if it has been
touched. The more often this current draw is measured, the better the responsiveness and fidelity of
the touch screen.
Second, by using your fingertip to interact with the
screen, you are less likely to push too hard on the
screen and damage it. Resistive digitizers require
that you exert a fair amount of pressure on them
to register a touch. Capacitive digitizers are only interested in the ability of your finger to change the
electrical current being sent through the screen.
Most customers I’ve talked to tend to err on the side
of being too gentle with their tablets, rather than
pressing too hard on them. It’s only natural, since
the history of electronics has been that devices are
too fragile. This reputation is such that users are
quite timid when it comes to using their fingers to
touch the screen.
In Conclusion
It’s an exciting time to be a mobile PC developer.
Device form factors are changing and the number
of ways to interact with a mobile PC is increasing.
All of which leads to a much more natural experience when using computing devices, whether you’re
writing notes in ink, or reading on the beach with
your Ultra-Mobile PC.
With just a few small changes in the way you think
about user interaction, you can develop apps that
work anywhere, anytime, and on any device. Think
of the possibilities!
Hilton Locke
Active, or capacitive technology when referring to
touch input, uses the capacitive properties of your
fingertip to determine touches. It works by applying a small amount of voltage to the touch screen.
When your fingertip touches the screen, a minute
amount of the current is drawn to the point of contact. The digitizer can then calculate the X and Y
coordinates of the touch.
Keep in mind that capacitive technology is different
from the electromagnetic technology in use in most
of today’s Tablet PCs. Electromagnetic digitizers
generate a small electromagnetic field and rely on
detecting the special electronic stylus’s disturbing
that field to determine the stylus’ position. There are
even some digitizers that combine capacitive and
electromagnetic technologies to create a single system device, for simplicity.
The advantages of capacitive touch digitizers over
resistive digitizers are twofold.
First, they have a higher sampling rate than resistive
digitizers. Resistive digitizers require measurement
of some mechanical interaction to detect touches.
This means that there is no real sampling rate for re-
56
Inking in ASP.NET 2.0,
AJAX, and IE7
In the past year, new technologies from Microsoft have changed
how we can add ink to Web sites and the change is definitely for
the better!
O
ne small property added to ASP.NET 2.0 servtrol while it is being designed. To test more complex
er controls, OnClientClick, has had a big imfunctionality, you may want to embed the control in
pact on simplifying the process of moving ink
a simple Windows Form where you can debug from
from the Web page to the Web server. The capability
the Windows Form code into the user control as
that OnClientClick provides,
long as their projects are in the
allowing developers to tie both
same solution.
Fast Facts
a client-side event and a serverASP.NET 2.0 and AJAX
side event to one Click event,
The second big difference is that
have made building inkwas actually achievable prior to
your Web application cannot acASP.NET 2.0 using control atcess the control’s methods from
enabled Web applications
tributes, but it was more commanaged code. Why not? Bemuch simpler.
plex to set up and not very discause the user must download
This article will walk you
coverable.
the embedded control to their
through building your
computer, you will not be able to
first ink-enabled control
Microsoft’s ASP.NET AJAX
interact with the control from the
(specifically designed to be
has removed the pain and the
server-side code, but only from
used in AJAX-enabled Web
complex coding required to enclient-side code—JavaScript.
sure that any ink in the control
sites) and leveraging the
remained in place during a page
This means that you must try to
new tools to get the control
postback. However, it does reembed as much of the control’s
integrated with your Web site.
quire a small change to how
functionality and logic within
You’ll also see how to prevent
you build your control.
the control and expose only the
IE7 from spoiling the user’s
most minimal amount of funcinking experience.
In addition to these enhancetionality to the calling code (the
ments, there are three other
JavaScript). For example, you’d
points to be aware of. The first
place all functionality for the ink
is the COM-Visible parameter for any ink controls
style (color, transparency, width) into the control.
that you will use in Web sites and the second is a
That way, you won’t have to write any JavaScript
change in Internet Explorer, which impacts how you
to attempt to deal with the colors. The only funcembed the ink control into the Web page. Lastly,
tionality that you should need to expose is anything
Microsoft Visual Studio® 2005’s new UserControl
that the page or Web application must control. For
TestContainer interface makes it much simpler to
example, if the page of the Web application must
test and debug User Controls. You no longer need
have knowledge of the color that the ink uses, you
to create a Windows Form just for this purpose.
would need to make a public property. As another
example, suppose you need to persist the ink data.
In this article, you will learn how to build a simple
That would require having a public method that reink-enabled Windows Forms control that is ready
turns the ink data to the calling code. The page’s
to be placed in an ASP.NET page (with or without
JavaScript function could call
AJAX) and how to write ASP.NET 2.0 and AJAXthat method and have access
enabled Web pages that can host this ink control.
to the ink data, and then pass
This
it onto the server-side code,
which could do something like
must try to
Creating an Ink-Enabled Control
store the data into a database.
When you create an ink-enabled Windows Forms
control to use on a Web page, you need to consider
two things that are different from a control that
you’d place in a Windows Form. First, you won’t be
able to debug the control from a Web site. Luckily,
the new UserControl TestContainer in Visual Studio 2005 allows you to run and debug the user con-
The ink control that I will create will have the following capabilities:
1. Draw in a variety of colors: black, blue, red, and
green.
Julia Lerman
[email protected]
Julia Lerman is an independent
consultant who has been
designing and writing software
applications for over 20 years.
She lives in Vermont where
she runs the Vermont.NET User
Group. Julia is well known in the
.NET community as an INETA
Speaker, .NET MVP, ASPInsider,
conference speaker, and prolific
blogger. You can read Julia’s
blogs at thedatafarm.com/blog
and on DevSource.com.
means that you
embed as much of the
control’s functionality and logic
within the control and expose
only the most minimal
amount of functionality to the
calling code (the JavaScript).
Inking in ASP.NET 2.0, AJAX, and IE7
57
• Add an InkOverlay component to the control.
• Instantiate and enable the InkOverlay.
The control’s code should look like this when you
have done these things:
VB
Imports Microsoft.Ink
Public Class UserControl1
Figure 1: The ink-enabled user control with the ToolStrip for
affecting the properties of the pen.
Eliot Graff
Manager, Developer
Experience, Mobile & Tailored
Platforms Division
The great thing about Windows
technology for mobile computers
is how powerful it is when it’s
paired with ubiquitous WiFi.
Nobody’s tethered to their desk;
nobody’s tethered to any desk!
2. Erase strokes.
3. Inform the Web page of the current color that
is being used.
4. Expose the current ink data in the control.
5. Load ink data into the control (with a special
tweak for AJAX pages).
6. Clear all ink from the control.
Private inkO As InkOverlay
Private Sub UserControl1_Load(ByVal sender As _
System.Object, ByVal e As System.EventArgs) _
Handles MyBase.Load
inkO = New InkOverlay(Me)
inkO.Enabled = True
End Sub
End Class
C#
To create this control, start with a Windows Forms
Control Library project, which will automatically
create UserControl1.
As with any Tablet PC development, you will need
to start with the following actions:
• Add the Microsoft.Ink API into the project
references.
using Microsoft.Ink;
public partial class UserControl1 : UserControl
{
private InkOverlay inkO;
public UserControl1()
{
Listing 1: VB and C# code for Public methods of the ink user control
VB
Public ReadOnly Property CurrentColor() As String
Get
Return inkO.DefaultDrawingAttributes.Color.Name
End Get
End Property
Public Function GetInkData_64() As String
If inkO.Ink.Strokes.Count = 0 Then
Return "empty"
Else
Dim inkBytes As Byte() = inkO.Ink.Save(PersistenceFormat.Gif, _
CompressionMode.NoCompression)
Return Convert.ToBase64String(inkBytes)
End If
End Function
Public Sub LoadInkData_64(ByVal inkData As String)
Dim inkBytes As Byte() = Convert.FromBase64String(inkData)
If inkO IsNot Nothing Then
inkO.Dispose()
inkO = New InkOverlay(Me)
inkO.Enabled = True
End If
inkO.Ink.Load(inkBytes)
Me.Invalidate()
End Sub
C#
58
public string CurrentColor
{
get { return inkO.DefaultDrawingAttributes.Color.Name; }
}
public string GetInkData_64()
{
if (inkO.Ink.Strokes.Count == 0)
{
//test for data
return "empty";
}
byte[] inkBytes = inkO.Ink.Save(PersistenceFormat.Gif,
CompressionMode.NoCompression);
return Convert.ToBase64String(inkBytes);
}
public void LoadInkData_64(string inkData)
{
byte[] inkBytes = Convert.FromBase64String(inkData);
if (inkO != null) //Ink overlay already exists
{
inkO.Dispose();
inkO = new InkOverlay(this);
inkO.Enabled = true;
}
inkO.Ink.Load(inkBytes);
this.Invalidate(); //forces a redraw
}
}
private void UserControl1_Load(object sender,
EventArgs e)
{
inkO = new InkOverlay(this);
inkO.Enabled = true;
}
}
In the control’s design surface, add some functionality for controlling the color.
Drag a ToolStrip control onto the UserControl.
You can create as many color buttons as you wish
by clicking the button icon on the toolstrip. Figure
1 shows the four new buttons after changing the
DisplayStyle to None and the BackColor to represent the desired colors.
I added two more buttons, which both have the DisplayStyle property set to Text. The first button’s Text
property is “Eraser” and the second is “Clean All”.
The ink control will contain all of these functions.
There’s no need to write any code in the Web page
to control these actions.
The buttons’ Click events will perform the following functionality:
• The four color buttons will cause the ToolStripColorButtons_Click event to fire, changing the pen color to the background color of
the clicked button.
• The eraser button will change the pen’s EditingMode to eraser. The default eraser mode
is StrokeErase, which causes the entire stroke
to be affected. Alternatively, you can change
the InkOverlay’s EraserMode to PointErase if
you prefer.
• The Clean All button cleans all ink from the
entire control.
Note that if you are new to C# Windows Forms,
you can wire up the Click event of each of the buttons to the following code in the Events portion of
the control property window. All four color buttons
will point to the btnColor_Click method.
VB
Private Sub btnColor_Click _
(ByVal sender As System.Object, ByVal e As _
System.EventArgs) Handles btnBlack.Click, _
btnBlue.Click, btnGreen.Click, btnRed.Click
Dim btn As ToolStripButton = sender
inkO.DefaultDrawingAttributes.Color = _
btn.BackColor
'make sure you are not in erase mode
inkO.EditingMode = InkOverlayEditingMode.Ink
End Sub
Private Sub btnEraser_Click(ByVal sender As _
Handles btnEraser.Click
inkO.EditingMode = InkOverlayEditingMode.Delete
End Sub
Private Sub btnClean_Click(ByVal sender As _
Handles btnClean.Click
inkO.Ink.DeleteStrokes()
Me.Invalidate() 'forces a redraw
End Sub
C#
private void btnColor_Click(object sender,
EventArgs e)
{
ToolStripButton btn = ((ToolStripButton)sender);
inkO.DefaultDrawingAttributes.Color =
btn.BackColor;
//make sure you are not in erase mode
inkO.EditingMode = InkOverlayEditingMode.Ink;
}
Michael Coulson
private void btnEraser_Click(object sender,
EventArgs e)
{
inkO.EditingMode = InkOverlayEditingMode.Delete;
}
With more end users using 3G
connections with their mobile
PCs, it’s important to optimize
applications to reduce chattiness
and enable seamless transitions
from online and offline states.
Business Development
Manager, Mobile & Tailored
Platforms Division
private void btnClean_Click(object sender,
EventArgs e)
{
inkO.Ink.DeleteStrokes();
this.Invalidate(); //forces a redraw
}
The last three functions, shown in Listing 1, will be
public so that the Web page’s client-side code can
interact with the user control. The first returns the
current color name, the second returns the ink data
as a Base64-encoded string, and the third will load
data into the control. This conversion to Base64 is
critical for enabling the ink data to be transmitted
over HTTP to get to the server-side code where you
might do something such as store the data into a
database, or in the case of this article, store it into
a file.
The LoadInkData_64 method has been written specifically to accommodate the control being placed
in an AJAX page. On a normal page, any time the
page posts back, the ink control would be re-instantiated. In that case, loading the ink data is simple.
However if you are using AJAX and the page is only
doing a partial postback where the ink control is
NOT part of that postback, you’ll have to load ink
into a control that may already have ink data in it.
This code does that by disposing and then reinstantiating the InkOverlay.
You can test the control by pressing F5 and invoking Visual Studio 2005’s UserControl TestContainer. As shown in Figure 2, you will find that the
CurrentColor property is visible in the properties
window and that you can use all of the functionality
on the control’s surface. However, you cannot test
the public methods here.
Web Site Limitations
You can build ink-enabled Web
sites using any of the Web
creation options in Visual Studio
2005. If you choose “Create new
web site” you can then create
the Web site either in the File
System or in HTTP. You can also
use a Web Application Project
either using the File System
Development server or HTTP/IIS.
Note that you will run into some
problem with File System sites
if you attempt to modify the
user control after the Web site
has first displayed it. See Tricks
and Traps for the reason and
workaround for the problem.
59
This completes the ink object. Once you have created the Web site, you will need to copy the DLL from
this compiled project into the Web site project.
Creating the Web Page to Host the Ink
Control
The page you are creating can:
• Avoid the embedded object “please click here”
problem in Internet Explorer 7.
• Enable partial postbacks with the ASP.NET
AJAX UpdatePanel.
• Send the ink control’s data to an image file.
• Retrieve ink data from an image file and load
it into the ink control for additional drawing.
• Display the current ink color being used in a
small panel.
Before creating this Web site, you must install
Microsoft’s ASP.NET AJAX. You can learn more
about AJAX and download the installer at http://
ajax.asp.net.
Figure 2: Running the UserControl project in Visual Studio 2005’s TestContainer.
Links
Workarounds for embedded
object issue from Microsoft,
“Activating ActiveX Controls”
[http://msdn.microsoft.
com/library/default.asp?url=/
workshop/author/dhtml/
overview/activating_activex.
asp]
Adobe‘s article,
“Active Content Update Article”
[http://www.adobe.com/
devnet/activecontent/articles/
devletter.html]
One last little design action to take is to set the
control’s BorderStyle property to FixedSingle. This
makes it easier to identify the boundaries of the
control when you embed it into the Web form.
Before building the project, it is necessary to change
the assembly’s ComVisible attribute. You can do
this through the project property pages by clicking
the Assembly Information button on the Application tab and checking Make assembly COM-Visible. You can also change this attribute directly in
the project’s AssemblyInfo file.
Figure 3: Copying the user control’s DLL to the Web site.
60
Also, note that you must create this Web site in
HTTP rather than using ASP.NET 2.0’s Development Web Server (file). Some of the functionality of
the embedded control does not work in the Development Web Server.
Create a new AJAX-enabled Web site. For the sake
of this demo, you will find it easier to do this in the
same solution where you created the user control.
Add the User Control to the Page
First you want to get the user control onto the page.
This is where you need to code around a “feature”
Figure 4: Using the ink control in an ASPX page with no client
or server-side coding.
of Internet Explorer 7, which is included in recent
updates to Internet Explorer 6, as well. The feature
requires that users explicitly choose to activate any
objects embedded into a Web page. So, users have
to choose to activate things like the Adobe® Flash®
object, Microsoft Windows Media Player, or any
ActiveX® control. Essentially they have to activate
on the page embedded using <EMBED> or <OBJECT> tags. Both Microsoft and Adobe have articles with workarounds for Web developers. See
the Links sidebar for more information.
Notice the pattern of the classid: AssemblyName.
dll#FullyQualifiedClassName and represents the
project name and the namespace plus class name
of the user control, since none of the defaults were
changed.
Now you need to call this JavaScript from the client
side. In the HTML source of the page, place the following in the spot where you would like to see the
ink control:
<script src=EmbedInkControl.js></script>
Locate the user control’s DLL in the first project.
You may have to Show All Files to see it. Copy the
file and paste it into the Web site project, just like
you might do for an image.
Use a Jscript file from the installed templates in
Visual Studio to dynamically embed the user control. Create a new JScript template file, named EmbedInkObject.js, and copy the following JavaScript
into it, removing any line wrapping in the actual
code:
document.write('<object id="InkControl" classid=
"CoDeFocusInkControl.dll#CoDeFocusInkControl.
UserControl1" height="400" width="400"/>');
Because the control is dynamically embedded at run
time, you will not see any evidence of it on the design surface of the page. Alternatively, you can place
the <object> tags and code directly in the HTML
for the sake of positioning and sizing the control,
but then use the <script> tags when you are ready
to deploy the page.
If you run the form now, you will see that all of
the control’s embedded functionality for drawing,
changing the pen color, erasing, and cleaning the
control work with no additional effort. Figure 4
shows the ink control in use before you add any
code to the Web site.
Tricks and Traps
Problem: You have embedded
the control in the Web page and
it looks great when running the
page. But when you try to get
it to run any of its methods, an
error “Object does not support
this method” is thrown.
Solution: Most likely, this means
that you did not set the control
assembly’s COM-Visible property
to True. Do this in the control’s
Project Properties.
Problem: At run time, the control
shows on the Web page as an
empty box with none of the
UI that you designed into the
control.
Solution: You probably have a
typo referencing the class in the
EmbedInkControl.js file. Watch
for casing also, as this is casesensitive.
61
1. Drag an HTML Input button control onto the
form. You can check in the source code to be
sure it is placed where you want it in relation
to the ink control.
2. Change the Value property of the button to
“Get Color”.
3. In the source of the input tags, add the OnClick parameter with the value “ShowCurrentColor()”.
4. Drag an HTML Div onto the form and change
its ID to colorDiv.
5. Place the following script in the source between the beginning <body> tag and the beginning <form> tag:
Problem: I tried out the control
in the Web site and it was fine.
Then I modified the control,
recompiled it, and copied the
new DLL into the Web site. Now
the control does not show up on
the page.
Solution: If you created your
Web site using the File System
server instead of HTTP/IIS, this is
a known problem with any user
controls, not just ink-enabled
controls. You will need to rename
the control each time you modify
and recompile for the Web page
to recognize the new version.
Since the file-based server is
for development only, this is a
design-time problem only.
Figure 5: Interacting with the control in client-side script.
<script>
function ShowCurrentColor()
{
document.all("colorDiv").style.backgroundColor=
form1.InkControl.CurrentColor();
}
</script>
6. Run the form again to see the button function
in tandem with the selected pen color as in
Figure 5.
Getting the Page to Interact
with the Control
Now that you have the functioning control on the
page, it’s time to get down to the real dirty work of
communicating with the control and getting the ink
data from it as well as back into it.
I’ll first show you how to get the CurrentColor information from the control. Since that information
is in the control on the client side, there is no need
for any server-side functionality, so you can do this
with a regular HTML input button and a DIV tag.
When a user clicks the button, this function fills the
DIV with the current pen color. Set it up following
these steps:
So far, interacting with the ink control is pretty
simple, especially when you only need to do so on
the client side.
Getting the Ink Data to the Server
Side—It’s Simple with OnClientClick
Now that you are an expert at interacting with the
embedded ink control, it’s time to go to the next
step: getting the ink data from the control and
sending it to the server side. Prior to ASP.NET 2.0,
this involved a lot of steps and trickery, but as mentioned above, the new OnClientClick property of
Listing 2: Method to persist the ink data in an XML file
62
VB
C#
Private Sub SaveasXMLFile(ByVal inkdata As String)
Using fs As New System.IO.FileStream( _
Context.Server.MapPath("")+ "\inkdataasxml.xml", _
System.IO.FileMode.Create)
Using xwriter As New System.Xml.XmlTextWriter(fs, _
System.Text.Encoding.UTF8)
With xwriter
.WriteStartDocument() 'create starting tag
.WriteStartElement("inkData")
.WriteElementString("Ink", inkdata)
.WriteElementString("LastUpdate",Now.ToString)
.WriteEndElement()
End With
End Using
End Using
End Sub
private void SaveasXMLFile(string inkdata)
{
using (System.IO.FileStream fs= new
System.IO.FileStream(Context.Server.MapPath("")+
"\\inkdataasxml.xml", System.IO.FileMode.Create))
{
using (System.Xml.XmlTextWriter xwriter=
new System.Xml.XmlTextWriter(fs,System.Text.Encoding.UTF8))
{
xwriter.WriteStartDocument(); //create starting tag
xwriter.WriteStartElement("inkData");
xwriter.WriteElementString("Ink", inkdata);
xwriter.WriteElementString("LastUpdate",
DateTime.Now.ToString());
xwriter.WriteEndElement();
}
}
some of the ASP.NET 2.0 server controls has simplified this enormously.
Follow three steps to accomplish this task.
1. Client Side: Get the ink data from the control.
2. Client Side: Store it in a hidden control that is
accessible from the server side.
3. Server Side: Read the data in the hidden control.
Once you have the data on the server side, you can
do whatever you wish with it. For example, store it
into a database, send it to a Web service, or save it
in a file. In this case, the example code will store it
into an image file on the local drive.
To get started, drag an ASP.NET HiddenField server control and an ASP.NET Button server control
onto the form. Change the button’s Text property to
“Get Ink to Server”.
In the OnClientClick property (visible in the property window) of the button, enter GetInkToServer().
C#
protected void Button2_Click(object sender,
EventArgs e)
{
string inkstr = HiddenField1.Value;
//now you have the data
//Save it to a GIF
byte[] bytes = Convert.FromBase64String(inkstr);
System.IO.File.WriteAllBytes(
Server.MapPath("") + "\\myimage.gif",bytes);
}
Getting Started
with AJAX
The Microsoft ASP.NET AJAX Web
site [ajax.asp.net] has the AJAX
downloads as well as articles and
tons of videos for getting started
with AJAX.
Run the project at its current state. In a default setup you will find the file, myimage.gif, in the project
folder inside “C:\inetpub\wwwroot\”.
And Now, AJAX
You may have noticed when running this last sample that when the page posted back after running
the server-side code, the ink control was empty.
That is because when the page posted back, it was
Copy the following script into the same section
where you put the ShowCurrentColor function:
Anything outside of the
UpdatePanel will be untouched
during the postback.
function GetInkToServer()
{
form1.HiddenField1.value=
form1.InkControl.GetInkData_64();
}
Back in design view, double-click the new button to
automatically create the server-side Click event of
the button. What’s perfect about the OnClientClick
for your purposes is that it runs before the serverside event code. This allows you to first place the
ink data into the hidden control, run the following
server-side code to grab that data, and then have
your way with it, so to speak:
VB
Protected Sub Button2_Click(ByVal sender As _
Object, ByVal e As System.EventArgs) _
Handles Button2.Click
Dim inkstr As String = Me.HiddenField1.Value
'now you have your hands on the data.
'Save it as GIF.
Dim bytes() As Byte = _
Convert.FromBase64String(inkstr)
System.IO.File.WriteAllBytes _
(Server.MapPath("") & "\myimage.gif", bytes)
End Sub
63
Listing 3: Two of the methods that are part of the code to load XML data into into the ink control
VB
C#
Private Function GetDatafromXML() As String
Dim inkData As String = ""
Using fs As New System.IO.FileStream(Context.Server.MapPath("") _
+ "\inkdataasxml.xml", System.IO.FileMode.Open)
Using xreader As New System.Xml. _
XmlTextReader(fs)
While xreader.Read
If xreader.Name = "Ink" Then
inkData = xreader.ReadString()
Return inkData
End If
End While
End Using
End Using
Return inkData
End Function
private string GetDatafromXML()
{
string inkdata;
using (System.IO.FileStream fs = new
System.IO.FileStream(Context.Server.MapPath("") +
"\\inkdataasxml.xml", System.IO.FileMode.Open))
{
using (System.Xml.XmlTextReader xreader = new
System.Xml.XmlTextReader(fs))
{
while (xreader.Read())
{
if (xreader.Name == "Ink")
{
inkdata = xreader.ReadString();
return inkdata;
}
}
return "";
}
}
}
Private ReadOnly Property LoadScript() As String
'creates the script that will be run when
' the UpdatePanel posts back
Get
Dim sb As New StringBuilder
sb.AppendLine("inkdata= " & _
"document.all('HiddenField1').value;")
sb.AppendLine(" if (inkdata!='')")
sb.AppendLine("form1.InkControl.LoadInkData_64(inkdata);")
sb.AppendLine(" document.all('HiddenField1').value='';")
Return sb.ToString
End Get
End Property
private string LoadScript
{
get {
System.Text.StringBuilder sb=new System.Text.StringBuilder();
sb.AppendLine("inkdata= document.all('HiddenField1').value;");
sb.AppendLine(" if (inkdata!='')");
sb.AppendLine("form1.InkControl.LoadInkData_64(inkdata);");
sb.AppendLine(" document.all('HiddenField1').value='';");
return sb.ToString();
}
}
rendered anew in its entirety. This means that the
ink control you see after the postback is a completely new control. Prior to AJAX, it was a complex chore to persist the ink and load it back into
the newly rendered control. But with AJAX, you
can get this to happen with very little effort.
Although you created an AJAX-enabled Web site,
you haven’t actually implemented any AJAX features yet! So now it’s time.
Figure 6: Moving the server
controls into the AJAX UpdatePanel
control.
1. In the design surface of the page, drag an AJAX
UpdatePanel control onto the page. The UpdatePanel along with the ScriptManager that
is placed on the page by default, will ensure
that only the contents of the UpdatePanel will
be involved in the postback. Anything outside
of the UpdatePanel will be untouched during
the postback.
2. Drag the Get Ink to Server button and the hidden field into the UpdatePanel as in Figure 6.
3. Run the form again. Draw something in the
control and click Get Ink to Server.
It seems as if nothing happened. The screen didn’t
flash and the ink remained in the control. But if you
64
look at the new myimage.gif, you will see that it has
the new drawing in it. For the true skeptic, you can
prove to yourself that something really happened by
putting a breakpoint in the Button2_Click event.
Congratulations, you are now an AJAX expert! Actually there is a lot more to learn about AJAX, but
there is clearly a lot of power just in the simplicity
of the UpdatePanel!
Loading Saved Ink into the Control
Once your application has saved the ink, whether it is someone’s doodles or perhaps some “red
line” markup on an architectural drawing, it’s
possible that the user or a colleague will need to
add to or modify the drawing. If your application
persisted the ink data properly, the application
can load it back into an ink control and continue
working.
The example above extracted the ink data, but when
it streamed the data into the GIF file, the application lost the actual ink data. But what if you had
stored it in a database field or an XML file? Then
the ink object’s structure is properly persisted and
you can retrieve it.
Add the code in Listing 2 to the code-behind of the
page. This will use an XMLTextWriter to build an
XML file that includes the ink data.
In the Click event of the Get Ink to Server button,
comment out the last two lines of code and add the
call to the new method as shown below:
VB
Protected Sub Button2_Click(ByVal sender As _
Object, ByVal e As System.EventArgs) _
Handles Button2.Click
Dim inkstr As String = Me.HiddenField1.Value
'now you have the data. Save it as GIF.
'Dim bytes() As Byte = _
' Convert.FromBase64String(inkstr)
'System.IO.File.WriteAllBytes _
' (Server.MapPath("") & "\myimage.gif", bytes)
SaveasXMLFile(inkstr)
End Sub
C#
protected void Button2_Click(object sender,
EventArgs e)
{
string inkstr = HiddenField1.Value;
//now you have the data. Save it to a GIF
//byte[] bytes=Convert.FromBase64String(inkstr);
//System.IO.File.WriteAllBytes(
//Server.MapPath("") + "\\myimage.gif",bytes);
SaveasXMLFile(inkstr);
}
Now, you will add functionality to pull that data
from the XML file back into the ink control.
Drag another standard button into the UpdatePanel and change its Text property to “Get Ink from
Server”. Double-click the button to get to its serverside Click event.
RegisterStartupScript(UpdatePanel1, Me.GetType, _
"PageLoadScript", LoadScript, True)
End If
C#
string inkdata;
inkdata = GetDatafromXML();
if (inkdata != "")
{
this.HiddenField1.Value = inkdata;
System.Web.UI.ScriptManager.
RegisterStartupScript(UpdatePanel1,
this.GetType(), "PageLoadScript", LoadScript,
true);
}
Next, add the GetDatafromXML and LoadScript
methods found in Listing 3.
Run the form again. Draw something in the ink
control and save it to XML with the Get Ink to
Server button. Clean the control so that it is empty
and click the Get Ink from Server button. The ink
is loaded back in. Modify the drawing and save it
again. Clean the control and load the drawing again.
You can see that you are now able to save, retrieve,
and modify the drawing. And thanks to AJAX, it all
happens very smoothly.
Now that you are familiar with the basics of using
the Tablet PC API and writing effective ink controls for ASP.NET as well as AJAX, you have the
tools to dig deeper into both technologies and wrap
them into more useful Web sites. There are a lot of
possibilities for online collaboration with ink, from
social applications such as signing birthday cards,
to business applications such as red-lining architectural drawings. Mobile computing becomes even
more mobile when you can take it to the Web.
Julia Lerman
Here you will leverage a few tricks. First you will
read the XML file data and store it into the HiddenField control. Then using the ClientScript Manager
specifically for AJAX, you will create some script
that will get included and run upon the postback.
This script will pull the data from the HiddenField
and pass it into the ink control’s LoadInkData_64
method.
Copy the following code into the server-side Click
event of the new button:
VB
Dim inkdata As String
'get ink from XML
inkdata = GetDatafromXML()
If inkdata > "" Then
Me.HiddenField1.Value = inkdata
System.Web.UI.ScriptManager. _
65
Basics of Ink in Windows
Presentation Foundation
Billy Hollis
[email protected]
www.dotnetmasters.com
(615) 333-6555
Billy Hollis is an author and
consultant specializing in .NET
technologies. He has written
several books and was coauthor
of the first book ever published
on Visual Basic .NET. He speaks
at major conferences worldwide,
including TechEd, VSLive,
DevConnections, and Microsoft’s
Patterns and Practices Architect
Summit events. He is available
for consulting and training on
.NET architecture, smart client
systems, Tablet PC, and WPF.
66
Ink is a first-class citizen in the next generation of Microsoft®
Windows® user interfaces.
Microsoft built ink functionality into all versions of the Microsoft Windows Vista™
operating system, and the new Windows Presentation Foundation (WPF) makes it easy
to ink-enable your applications. Even using ink over moving video is easy in WPF!
I
nk applications in WPF are similar in many reGetting Started
spects to those in Windows Forms. You need to
collect ink, display it over images, store and reIf you have worked with ink before, you’ve probtrieve ink, and perform recognition on ink to find
ably used Windows Forms or direct access to COM
out what it says. In this article,
objects for ink functions. WPF
I’ll look at the basics of performadds a new way to host ink
Fast Facts
ing each of these functions.
functionality.
Microsoft has refined many
Tablet PC ink capabilities in
For the examples, I assume that
The easiest way to get started
you are working on Windows
with ink in WPF is by using a
Windows Vista and Windows
Vista. It is possible to configure
new WPF control, the InkCanPresentation Foundation.
a Windows XP system to intevas. This control combines all
grate ink with WPF, and Eliot
the things you would do with
Graff talks about that in his article “The Proper
InkCollector, InkPicture, and InkOverlay conDeveloper Environments for Mobile PC, Tablet
trols when doing development on the classic TabPC, and Ultra-Mobile PC Applications,” elsewhere
let PC.
in this issue. In Windows Vista, Microsoft built in
everything needed to do ink in WPF, except for the
You’ll find the InkCanvas control in the System.
development tools.
Windows.Controls namespace, along with all the
other routine WPF controls. Other objects associYou will need to install both Microsoft Visual Stuated with ink manipulation are in the System.Windio® 2005 and the Visual Studio 2005 extensions
dows.Ink namespace, including objects for holding
for .NET Framework 3.0. You can download the
ink and doing recognition from ink.
extensions from the Microsoft Web site. Start at
http://msdn.microsoft.com/vista, and then look for
I’ll run through a simple example to show you just
the link for Windows Vista downloads.
how easily you can get started with ink in WPF.
After installing the Visual Studio extensions, start
The extensions include project templates for projVisual Studio 2005, and then select the option to
ects in WPF and in Windows Communication
create a new project. You’ll see a new subfolder
Foundation (WCF). This article is only concerned
for .NET Framework 3.0 project templates just bewith WPF, so I won’t discuss WCF any further.
neath the folder for Windows application project
templates. Click that folder to see the project types
If you’re not working on a Tablet PC and you also
available.
don’t have a digitizer attached to your system, that’s
OK. You can use your mouse as an ink input device
You’ll see a template for “Windows Application
on any Windows Vista system. All the examples will
(WPF)”. This is similar to a Windows Forms appliwork on a vanilla Windows Vista system.
cation, except that WPF is the UI technology used.
Select that template type, choose an appropriate
I’m not assuming that you know anything about
folder to store your project, and then enter a project
WPF. However, I don’t go very deeply into the WPF
name of “InkInWPF1”.
concepts. I only introduce a few basics that are necessary to start working with ink.
You’ll see a design surface (called a WPF Window),
which looks superficially similar to a form in WinI wrote the code examples in this article in Visual
dows Forms. It has a default name, Window1. Even
Basic®, and you only have to make minor changes
though it looks like a form, working with controls
in syntax to run them in C#. Only two examples
on the surface is different from the way it’s done
use code. Most of the examples use the markup lanwith a Windows Form. I won’t spend a lot of time
guage for WPF called Extensible Application Markexplaining how WPF positions things; others will
up Language (XAML). I’ll talk about that below.
write a complete article on that subject.
Basics of Ink in Windows Presentation Foundation
The visual designer for WPF in the Visual Studio 2005
Extensions is a placeholder until the next version of
Visual Studio is ready. That means it’s not as polished
as the Windows Forms designer. When working with
controls, you’ll need to work a bit harder to position
and size them than you did in Windows Forms.
to be on top of an image, as an annotation to the image. In WPF, that requires setting the Background
property of the InkCanvas.
Drag an InkCanvas control from the Toolbox onto the
window. You should position and size the InkCanvas
to be roughly the same size as the WPF window.
Notice that I said “Background” property, not
“BackgroundImage”. An InkCanvas doesn’t have
a BackgroundImage property. Instead, the Background property is a WPF brush, which means you
can base it on various visual elements, such as a
color, an image, or a gradient fill.
To make sure the designer doesn’t restrict the area in
which you can input ink, select the InkCanvas control and set the Margin property to all zeros. To do
this, right-click the Margin property and select Reset,
which sets the Margin property to its default value of
all zeros.
The visual designer for WPF at this point does not
let you set a background image for an InkCanvas
control in the Properties window. But you can edit
the XAML to specify a background brush based on
an image. Change the XAML for the InkCanvas to
look like this:
Press F5 to run the program. Use your stylus (or
mouse) to scribble in the window.
<InkCanvas Name="InkCanvas1">
<InkCanvas.Background>
<ImageBrush
ImageSource="C:\Dev\Images\Creek.jpg"/>
</InkCanvas.Background>
</InkCanvas>
Using XAML to Create Ink-Enabled
Surfaces
Now I’ll show you an alternate way to quickly get
an InkCanvas on a WPF window by editing the
XAML markup that Visual Studio uses to define
the window. You’ve probably already noticed that
the XAML shows up in an editor window just under
the Window1 design surface.
XAML is an XML-based markup language for declaring object trees. Since the visual elements that
make up a window are just such a tree, XAML is
one of the ways you can define a WPF Window.
XAML is very flexible, and many of the tasks you
might expect to accomplish in code are instead
done in XAML in WPF.
I’ll run through modifying XAML directly. Delete
your current Window1, and then select Project >
Add New Item. Select the item type of Window
(WPF) and click Add. You’ll get a new Window1
with no InkCanvas on it.
In the XAML for Window1, you’ll see the following
element:
<Grid>
</Grid>
Edit the XAML and replace this element with the
following:
Shawn Van Ness
Program Manager, Tablet PC
Platform Team
You’ll need to supply a valid image file name on
your system, of course. I copied the “Creek.jpg”
image from the Sample Pictures that Windows
Vista installs.
This is a typical example of how XAML allows
properties to be set for a WPF control. Because
the designer for WPF is still rather primitive,
I’ll do several of my examples using XAML instead.
Tablet PC technology is becoming
mainstream—you can see this
in a number of ways. The price
delta between Tablet PCs and
similarly configured laptops is
dropping; and Tablet PC features
are built into Windows Vista—
there's no special Tablet PC
Edition anymore. And last but not
least, components and controls
for Tablet PC applications are
built into our newer programming
frameworks, such as Windows
Presentation Foundation.
You can also change the ink color and width in
XAML. You’re probably familiar with the DefaultDrawingAttributes property of the InkPicture
control in Windows Forms. The WPF InkCanvas
control has a property with the same name and it’s
used the same way. You set the color and width
of ink by setting the Color and Width properties
of the DefaultDrawingAttributes for the InkCanvas. Here’s how the XAML would look if you enhanced the previous example to change the ink
color to blue and the ink width to 4:
<InkCanvas.Background>
<ImageBrush
ImageSource="C:\Dev\Images\Creek.jpg"/>
</InkCanvas.Background>
<InkCanvas.DefaultDrawingAttributes >
<DrawingAttributes
Color = "Blue" Width = "4" />
</InkCanvas.DefaultDrawingAttributes>
</InkCanvas>
</InkCanvas>
Now press F5 to run and notice that the InkCanvas
is functional, just as before.
Ink Over Media
Your applications will probably need more than
bare ink, of course. Many applications require ink
Up to now, I’ve looked at functionality that’s
about the same as what you could achieve in
67
Listing 1: Saving ink from an InkCanvas control to a file
Dim SaveFileDialog1 As New Microsoft.Win32.SaveFileDialog()
SaveFileDialog1.Filter = "isf files (*.isf)|*.isf"
If SaveFileDialog1.ShowDialog() Then
Dim fs As New System.IO.FileStream(SaveFileDialog1.FileName, _
System.IO.FileMode.Create)
InkCanvas1.Strokes.Save(fs)
fs.Close()
End If
Listing 2: Fetching ink from a file and placing it in an InkCanvas control
Dim OpenFileDialog1 As New Microsoft.Win32.OpenFileDialog()
OpenFileDialog1.Filter = "isf files (*.isf)|*.isf"
If OpenFileDialog1.ShowDialog() Then
Dim fs As New System.IO.FileStream(OpenFileDialog1.FileName, _
System.IO.FileMode.Open)
InkCanvas1.Strokes = New Ink.StrokeCollection(fs)
fs.Close()
End If
Using Code Instead
of XAML
The visual designer for WPF
creates XAML markup to hold the
visual design of a WPF window.
You can also create any WPF
visual element in code. Such
code looks similar to the code
for defining a form in Windows
Forms. It contains logic to create
visual controls and place them
in an appropriate container and
to set properties on controls as
necessary.
You could create a WPF
application with no XAML at all.
You can define the application’s
user interface entirely in code.
However, no designer yet exists
to create such a WPF program.
Microsoft created XAML to make
it easy to write cross-language
designers for WPF and to serve
as a cross-language declarative
format for defining WPF
interfaces. Most of the work you
do in WPF will involve XAML.
Windows Forms. Now I’ll move on to something
that’s close to impossible in Windows Forms—ink
over video.
Enabling ink over video is just as easy as placing
ink over an image because Microsoft built the basic functionality of media management into WPF.
The InkCanvas control simply has another property, called MediaElement, to specify the media you
want to use.
I took the video for my sample from the Sample
Videos folder installed with Windows Vista. Sample
Videos is available in the Videos folder, which is
another user folder. You can use your own video,
or you can grab the same video I used at “C:\Users\
{username}\Videos\Sample Videos\Bear.wmv”. In
my example, I’ve copied the “Bear.wmv” file to another directory to keep the path short.
To view this video, you’ll want to expand the size
of Window1. You should highlight Window1, and
then resize it to be about three times as wide and
twice as high as it was originally.
Change the XAML for your InkCanvas to look like
this:
<MediaElement Source="C:\Dev\Images\Bear.wmv" />
<InkCanvas.DefaultDrawingAttributes >
<DrawingAttributes
Color = "Blue" Width = "4" />
</InkCanvas.DefaultDrawingAttributes>
</InkCanvas>
Notice that I removed the brush used for the background image.
Now run the program and you’ll be able to write ink
directly on the video. It’s really that easy in WPF to
create a “chalk-talk” type of application!
68
Storing Ink
If you’ve worked with ink in Windows Forms,
you’re probably familiar with the Save method of
an Ink object. The Save method returns a stream of
bytes that represents the ink. The Load method can
take such a stream of bytes and recreate the ink.
Storing ink is a bit harder in WPF than Windows
Forms, but not much. I’ll show you an example in
which I save the ink that I drew over the bear video
above.
I’ll use code instead of XAML to save the ink because I need to respond to a user clicking a button.
WPF can attach a control’s events to code, and it’s
done the same way as in Windows Forms.
For my example, I’ll need two buttons for the user
to press. One will save the ink and the other will
restore it to the screen.
Drag two buttons from the Toolbox into Window1.
Position them close to the bottom of the window.
In the respective Properties windows, change the
first button’s Content property to “Save Ink”, and
then change the second button’s Content property
to “Restore Ink”.
Because of a glitch in the beta version of the designer, you’ll need to execute the program before
you add any code. (Otherwise, the buttons may not
show up in the drop-down list of objects that appears in the code window.) Right-click Window1.
xaml in the Solution Explorer, and then select View
Code.
In the code editor for Window1, click the left dropdown list and select Button1. Then click the right
drop-down list and select the Click event for Button1. You’ll get an empty Click event routine, just as
you would for Windows Forms.
Place the code from Listing 1 in the Click event.
The code displays a common dialog box and asks
the user to provide a file name. The Save operation
then places the bytes representing the ink into a
Enabling ink over video
is just as easy as placing
ink over an image.
That’s because Microsoft built
the basic functionality of media
management into WPF. The
InkCanvas control simply
has another property,
called MediaElement, to specify
the media you want to use.
stream object. (Notice that this is completely different from the equivalent Save operation in Windows Forms, which places the bytes into a byte array.) In this case, I’ll use a FileStream to receive
the bytes, but any kind of stream will do. If you
want to place the bytes in a database, for example,
you would probably use a MemoryStream to get
the bytes into a byte array, and then put them in
the database.
The Save method of the Strokes collection on the
InkCanvas places the bytes for the ink into the
stream. The FileStream then writes those bytes to
disk when the Close method runs.
I’ll set up the second button. In the code editor for
Window1, click the left drop-down list and select
Button2. Then click the right drop-down list and select the Click event for Button2. In the empty Click
event routine, place the code from Listing 2.
This code performs the inverse of the previous
Save Ink operation. This time, I’ll use a common
dialog box for opening a file that contains ink. The
FileStream based on that file feeds the bytes representing the ink into a new StrokeCollection, which
is then assigned to the Strokes collection of the InkCanvas.
Basic Ink Recognition in WPF
WPF handles ink recognition a bit differently from
Windows Forms on a Tablet PC. I don’t have enough
space to cover recognition comprehensively, but let
me show you an example that does basic recognition. First I need to add a reference to the DLL that
contains the InkAnalyzer class, which is the class
needed for ink recognition in WPF.
Right-click the InkInWPF1 solution, and then select Add Reference. Click the Browse tab, and then
browse to the needed DLL. Assuming you have a
typical system setup, WPF will name the file “C:\
Program Files\ Reference Assemblies\Microsoft\
Tablet PC\v1.7\ IAWinfx”.
Add another button to Windows1 from the Toolbox,
and then change its Content property to Recognize.
Create a Click event as you did for the earlier buttons, and then place the following code in the event
routine for the new button:
Dim MyInkAnalyzer As New _
System.Windows.Ink.InkAnalyzer()
MyInkAnalyzer.AddStrokes(InkCanvas1.Strokes)
MyInkAnalyzer.Analyze()
MsgBox(MyInkAnalyzer.GetRecognizedString)
Run the program, input some ink, and then click
the Recognizer button. You should get recognized
text in a message box, as in Figure 1.
There’s quite a bit more functionality available in the
InkAnalyzer object. You can get a tree structure of
potential recognized results for example. Microsoft
Figure 1: Ink entered over a moving video and then recognized. The message box contains the
recognized text.
streamlined the recognition functionality for ink in
WPF from that in Windows Forms, so I think you’ll
find it easier to use. Check the MSDN Mobile PC
Developer Center (msdn.microsoft.com/mobilepc)
and the article by Frank Lavigne, “Exploring Ink
Analysis” in this magazine for more information.
Conclusion
You have seen in this article just how nicely ink integrates with WPF, and how easy it is to ink-enable
your WPF applications. I’ve covered input of ink,
saving and restoring of ink, placing ink on top of
images and videos, and simple recognition.
I don’t have space to go over all the changes to ink
in Windows Vista, including better capability to position Tablet PC Input Panel, new auto-complete
capabilities on Input Panel, better ways to give hints
for ink recognition, and more control over events
on the stylus.
Billy Hollis
InkCanvas as a WPF
Container
In WPF, unlike Windows Forms,
there is a clear distinction
between controls that can contain
other controls and those that
cannot. All controls can contain
at least one element, referred to
as its “content”. Containers can
hold multiple child elements.
InkCanvas is a container which
may contain other containers.
If you examine the XAML that
resulted from dragging the
buttons onto a WPF Window in
this article, you’ll see that the
buttons are contained in the
InkCanvas.
A container can other containers.
Another example of a container
in WPF is a grid, and it’s possible
for an InkCanvas to be in a cell of
such a grid container. Moreover,
you can stack multiple elements
in a cell, so placing an InkCanvas
in a cell still allows other controls
in the cell to work properly.
Placing an InkCanvas in a grid
cell is a common technique
to restrict inking functionality
to certain parts of the user
interface.
69
Are You Ready To Start?
EPS
Can Help!
EPS Software provides:
Application
Development
•
•
•
•
•
.NET 3.0 (WPF, WCF)
Windows & Smart Client
ASP.NET & AJAX Web
Tablet PC & Mobile PC
Windows Mobile
Prototyping Services
Situation Assessment
Project Management
Mentoring
Training
Contact us at:
[email protected]
866-529-3682
www.eps-software.com
Exploring Ink Analysis
Frank La Vigne
[email protected]
Frank La Vigne is a Microsoft®
Tablet PC MVP and Lead
Architect/Designer for Applied
Information Sciences (AIS) in
Northern Virginia.
Frank started in software
development when he was
twelve, writing BASIC programs
for the Commodore 64.
He began his professional
career writing Visual Basic 3
applications for Wall Street firms
in 1993. He then moved on to be
the first webmaster for a major
book retailer. Frank then went on
to develop a large multinational
online banking project in
Germany.
In 2004, Frank became heavily
focused on Tablet PC application
development. Since then, he has
been working on various tabletbased solutions.
You can read more on his blog at
http://www.FranksWorld.com
The Tablet PC SDK makes it easy to incorporate digital ink
and handwriting analysis into applications; and now the
InkAnalysis API (available in Windows Vista™ as well as downlevel
to the Microsoft® Windows® XP operating system through a
redistributable) takes it one step further.
Actually, the InkAnalysis API exposes some of the lower-level functions that make
handwriting recognition possible. It also exposes some functionality that can improve
recognition results, support shapes, alternative recognition results, and spatial
analysis. In this article, I will take a deeper look into what goes on behind the scenes
and how to take advantage of the tablet team’s hard work.
W
hen I first started to work on Tablet PC soluit is with handwriting recognition. As you develop
tions, the depth and the richness of the Tablet
more Tablet PC applications, you will quickly find
PC API immediately impressed me. Thanks
that the ToString method of handwriting recognito the well-designed Tablet PC
tion leaves you wanting more.
SDK, I could add handwritFast Facts
ing recognition and support for
What if you wanted to see more
digital ink without concerning
than just the top result? What if
The Ink Analysis API
myself with the gritty details and
you wanted to recognize shapes
goes beyond text recognition
hard work of handwriting recas well? What if, in addition to
and interprets the structure
ognition.
converting the text of the docuof a document adding more
ment, you wanted to copy the
power and
As far as most tablet application
structure of the document?
value to ink-enabled
developers are concerned, handapplications.
writing recognition is as easy as
Ink analysis makes all this and
calling a ToString method.
more possible.
The following code best exemplifies the basic scenario of converting ink to text that most tablet developers are familiar with.
But first, I will briefly review the basic mechanics of
digital ink and what goes on when your code calls
the ToString method.
string recognitionResults =
inkCollector.Ink.Strokes.ToString();
The Basics of Recognition
MessageBox.Show("Recognition Results:" +
recognitionResults);
As you can see in the above code, the Strokes collection’s ToString method does all the work here,
easily converting the ink strokes to text.
But, is handwriting recognition really that easy? Of
course it isn’t. A lot of research went into building
the handwriting recognition engine to make it this
easy for tablet developers.
What goes on behind the scenes? More to the
point, is there any functionality
exposed at these lower levels of
the API? How can you leverage
the
the process to add advanced
functionality to applications?
narrowing
You can assist
recognition engine by
the vocabulary to a
particular word list.
72
Developers are never happy for
long. They have a curious nature
and push the envelope—to keep
expanding what’s possible. So
Most developers know that the Tablet PC can convert handwriting to text, but fewer know about the
shape recognition features of the Tablet PC platform. The subject of ink analysis covers handwriting, shape, and document recognition. First, I will
examine the path ink takes when you call the ToString method.
Recognizing Handwriting
The ink data type is a vector-based graphics format
that consists of a series of strokes. Strokes are Bezier curves, which are made up of a series of discrete
points.
The handwriting recognition engine analyzes these
strokes, produces a list of possible alternatives comparing them against a system-wide dictionary, and
ranks them according to what the engine believes
is the most likely result. The result with the highest confidence rank wins and becomes the value
returned from the recognizers.
In order to do this, the handwriting recognition engine has to determine several important key factors;
the same factors that the human eye takes into account when reading text.
First, the engine performs a rudimentary spatial
analysis on the ink: determining the baseline on
which the text is written, the boundaries of paragraphs, lines, and even individual words. The analysis in this scenario is quite different than the work
done in the InkAnalysis API, which I will dig deeper
into later in this article.
Figure 1 shows the baseline drawn out over a line
of text. The spatial analysis engine then passes
these individual words to the recognition engine,
which evaluates them and returns a text value and
a confidence level. The recognition engine assigns
a confidence level to every text result it associates
with a series of strokes. It returns the result with the
highest level of confidence as the value from the
ToString method in the previous example.
Meet the InkAnalyzer
The InkAnalyzer class contains all the methods,
properties, and events required to perform ink analysis. Experienced Tablet PC developers may think
of the InkAnalyzer as a new and improved version
of the RecognizerContext. The InkAnalysis object
contains every API the RecognizerContext has. In
short, the InkAnalysis object does everything the
RecognizerContext does and more.
The InkAnalyzer constructor class accepts two parameters: one is the Ink object it will collect ink
from and the other is the control associated with the
InkAnalyzer. After initializing the object, I call the
AddStrokes method to add strokes to the analyzer.
Finally, I will make a call to the Analyze method,
which performs the analysis.
The following code demonstrates how to initialize
an InkAnalyzer object.
InkAnalyzer analyzer = new
InkAnalyzer(this.collector.Ink,this);
analyzer.AddStrokes(this.collector.Ink.Strokes);
At first it may seem redundant to call the AddStrokes
method to pass the Strokes collection into the InkAnalyzer, when it already received the corresponding Ink object in the constructor. Individual stroke
objects in the Strokes collection each have a unique
identifier. Adding strokes from more than one Ink
object could potentially mean that two strokes
might have the same ID. This situation would confuse the InkAnalyzer. That is why the InkAnalyzer
will throw an exception if I add Strokes from different InkCollectors.
Additionally, the Save and Load methods for the
InkAnalyzer persist the analysis results along with
the ink. Therefore, the InkAnalyzer needs a reference to preserve the ink.
Figure 1: The recognition engine first determines the baseline on which the text is written.
To actually analyze the ink, I simply call the Analyze method.
analyzer.Analyze();
The InkAnalyzer contains within it all the functionality of RecognizerContext. However, you will need to
refactor code in order to perform the same tasks with
the InkAnalyzer, due to some architectural changes.
In the following sections, I will demonstrate how
to perform some of these basic tasks with the InkAnalyzer.
Improving Recognition Results
The obvious disadvantage of using a system-wide
dictionary to compare ink against is that no dictionary, no matter how complete, can contain all the industry-specific jargon, internal company references,
or even slang that users might use.
There is also the matter of fields expecting certain
types of data. For instance, if an entry field is expecting to only have dates,
then you can tell the recognition engine to have a bias toIn short, the InkAnalysis
wards converting the ink into
a date.
object does everything the
Hints and Factoids
RecognizerContext
does and more.
You can help the recognizer
achieve more accurate results
by setting the Factoid property of the AnalysisHintNode object to predefined values in an enumeration.
Using factoids, you narrow down the possible result
set as the recognizer interprets the strokes. If you do
tell the recognizer to expect an e-mail address, then
it is far more likely to interpret the strokes that represent the “@” symbol as “@” and not as a poorly
drawn letter “O.” The following code does just that:
it tells the recognizer that it should consider the
strokes to be an e-mail address.
AnalysisHintNode hintNode =
inkAnalyzer.CreateAnalysisHint();
hintNode.Location.MakeInfinite();
hintNode.Name = Factoid.Email;
73
The call to hintNode.Location.MakeInfinite() tells
the AnalysisHintNode that it should apply the Hint
across the entire InkCollector. I could choose to
narrow down the area to a particular region.
A good example is an e-mail client, where the topleft corner is an ideal place to write an e-mail address. I can create an AnalysisHintNode and restrict it to a set of coordinates.
Figure 2: The Tablet PC Input
Panel offers alternative recognition
results and an easy means to
select them.
Rectangle hintRegion = new Rectangle(0, 0, 4000,
4000);
analyzer.CreateAnalysisHint(new
AnalysisRegion(hintRegion));
hintNode.Factoid = Factoid.Email;
The ability to narrow down the area that an AnalysisHintNode can influence comes in handy when
designing applications that mimic paper forms. I
can tell the analyzer to expect certain types of data
in certain areas: for example, an e-mail address in
the upper-left corner and a date in the upper right.
Then I can define areas for numbers and currency
values.
I can also chain factoids together. The following
code tells the recognizer to expect either an e-mail
address or a Web address.
hintNode.Name = Factoid.Email + "|" + Factoid.Web;
Alternatively, you can also use string constants to
set the Factoid value. For instance, the following
code snippet is functionally equivalent to the above
snippet.
hintNode.Name = "EMAIL|WEB";
The Factoid class has values for dates, numbers, file
names, and much more—enough to cover virtually
every data entry need you will come across.
However, what happens when the data your code
expects doesn’t fit neatly into the category of date,
number, or e-mail address? For these situations,
you can create custom word lists.
Custom Word Lists
By default the recognition engine compares strokes
against the generic dictionary for analysis. However, this may not be appropriate for certain situations
as each industry has its own jargon. For instance,
medical specialties will have different jargon from
field insurance agents or sales teams.
You also might encounter a situation where you
want to limit the recognition results to a certain set
of words, yet still keep the input field a freeform ink
field. In these situations, use a word list.
I can assist the recognition engine by narrowing
the vocabulary to a particular word list. I do this
by creating a string array, populating it with words,
and then applying the word list via the SetWordList
method and setting Factoid properties appropriately. Note that WordList is a type of Factoid.
The following code demonstrates how to accomplish this by telling the recognizer to expect certain
medical terms.
string[] wordListArray = { "cereberal",
"hyperbaric", "hypoglycemic", "anemic" };
inkAnalyzer.CreateAnalysisHint();
Figure 3: Sample document before ink analysis in OneNote 2007.
Listing 1: Handling a gesture recognition event
void inkCollector_Gesture(object sender,
InkCollectorGestureEventArgs e)
{
if (e.Gestures[0].Id == ApplicationGesture.Circle)
{
if (e.Gestures[0].Confidence == RecognitionConfidence.Strong)
{
MessageBox.Show("I am very certain you drew a circle
Gesture.");
}
if (e.Gestures[0].Confidence ==
RecognitionConfidence.Intermediate)
{
MessageBox.Show("I am reasonably certain you drew a
circle Gesture.");
}
}
}
74
hintNode.Location.MakeInfinite();
hintNode.SetWordlist(wordListArray);
hintNode.Name = "Wordlist";
You could also create lists consisting of part numbers,
office locations, or virtually any discrete list of string
values. The source data for custom word lists can
come from a database, Web service, or XML file.
Alternate Recognition Results
No matter how smart the recognition engine becomes or how much code can anticipate user input,
you can never guarantee that any application will
achieve recognition results of 100%.
However, you can use the Tablet PC Input Panel,
which ships as part of the Windows XP Tablet PC
Edition 2005 and Windows Vista, to provide an easy
means for correcting the recognition engine. Tablet
PC Input Panel, as seen in Figure 2, has an excellent
user interface for the quick and easy modification of
understand formatting and structure cues, such as
paragraphs, bulleted lists, and annotations.
Accordingly, they, as human beings, have a level of
context and understanding on top of the actual content of the document.
The InkAnalyzer gives the operating system of a
Tablet PC the power to parse and understand this
organizational structure. Other ink-enabled applications, such as Microsoft OneNote® 2007, also leverage this ability. The ink in Figure 3 becomes the
structured text in Figure 4.
Figure 4: Sample document after ink analysis in OneNote 2007.
recognized text. So how do you go about adding this
type of feature to your own applications?
Fortunately, the code to access the alternate recognition results is quite straightforward, as you can
see in the following example.
Analyzing the Organizational
Structure
The structure of a document contains a lot of information. Thanks to the InkAnalyzer, you can now
analyze and parse it into a tree structure.
I recursively parse the structure the InkAnalyzer
created by starting with the RootNode property.
string msgText = "Matches & Confidence Levels:\n";
TraverseNodes(inkAnalyzer.RootNode, sbNodes, 0);
foreach (AnalysisAlternate recoAlternates in
inkAnalyzer.GetAlternates())
{
msgText += recoAlternates.RecognizedString + "
(" + recoAlternates.InkRecognitionConfidence.
ToString() + ")\n";
}
MessageBox.Show(msgText);
This code iterates through the collection returned
from the GetAlternates method of the InkAnalyzer. Each of the AnalysisAlternate objects has both
a text value and a confidence level.
Most of the time, the recognition engine will return
the correct result. However, you can use code similar
to the snippet above to build out a list of alternate
recognition results and display them to the user.
Introducing Ink Analysis
Once a collection of ink goes beyond several lines,
the ink ceases to be mere text and starts to become
a document, complete with paragraphs, bulleted
lists, callouts, and diagrams.
Listing 2 shows the complete contents of the TraverseNodes methods.
As the InkAnalyzer processes the strokes, it creates
a tree structure consisting of ContextNode objects.
The tree starts at the top with the RootNode, and
then progressively represents smaller and smaller
collections of Strokes. These nodes can consist of
WritingRegions, Paragraphs, Lines, Drawings, Individual Words, and more.
Qi Zhang
Development Lead,
Handwriting Recognition
Group, Mobile & Tailored
Platforms Division
Handwriting recognition enables
natural input mode for all
languages and is particularly
beneficial to East Asian
languages where characters
are composed of strokes, not
alphabets. My team and I are
very excited to work in this area
and to deliver the world-class
recognition technology to our
East Asian customers.
I can examine the particular type of ContextNode
with the following line of code.
contextNode.GetType().ToString()
Figure 5 shows both the original ink on the screen
and the contents of the ContextNode tree.
Deeper Dive into InkAnalysis
The InkAnalyzer engine parses a block of ink and
creates a tree structure. The InkAnalyzer can also
Listing 2: TraverseNodes method
Beyond Handwriting Recognition to
Document Recognition
The structure of the written word is just as important as the contents. There is a lot of semantic richness embedded into the structure and layout of a
document.
Users hardly notice this structure in their day-today lives, as it’s so familiar to them. They natively
private void TraverseNodes(ContextNode contextNode, StringBuilder
nodes, int Depth)
{
nodes.Append(string.Empty.PadLeft(Depth * 4, ' '));
nodes.Append(contextNode.GetType().ToString() + " ");
nodes.Append("[" + contextNode.Strokes.ToString() + "]");
nodes.Append("\r\n");
foreach (ContextNode subNode in contextNode.SubNodes)
{
this.TraverseNodes(subNode, nodes, Depth + 1);
}
}
75
Michael Bernstein
Lead Software
Development Engineer,
Tablet PC Team
My team works on making
handwriting recognition more
accurate, flexible, and available
in an increasing number of
languages. I’ve been working
on Tablet PCs for five years,
and I love seeing a completely
new kind of PC coming to be,
especially one that I wish I’d
had in college! My favorite
new Windows Vista feature is
Handwriting Personalization: the
Tablet can now learn my own
handwriting and vocabulary,
which makes working with
the pen a much smoother
experience.
Figure 5: The ContextNode tree represents the structure
inferred from analyzing strokes of ink.
Figure 6: Ink sample with numbered list items.
recognize bulleted and numbered lists. Table 1
shows the complete analysis of the ink shown in
Figure 6.
Relationships between Nodes
Recognizing Shapes
The InkAnalyzer can also recognize shapes. Table
2 has all the types of shapes it can recognize. While
the analyzer identifies shapes in the ink, it is difficult
to actually know what shapes it can recognize. The
API does not contain an enumeration of shapes, so
the listing in Table 2 will come in handy.
ContextNodeCollection drawingNodes =
inkAnalyzer.FindNodesOfType(ContextNodeType.
InkDrawing);
string results = "Drawings detected:\n";
foreach (InkDrawingNode drawingNode in
drawingNodes)
{
results += drawingNode.GetShapeName() + "\n";
}
MessageBox.Show(results);
The above code creates a ContextNodeCollection,
consisting only of InkDrawingNodes. I then iterate
through the collection and report the results in a
MessageBox.
Written notes often contain callouts, notations, and
underlines. When the InkAnalyzer detects these
items, it creates ContextLinks. ContextLinks describe relationships between nodes. ContextLinks
have two items of interest: the SourceNode and the
DestinationNode.
For example, in Figure 7, the word “this” has an underline. The InkAnalyzer knows that it is not part of
the word, but it is an annotation adding emphasis.
The following code iterates through the ContextLink
collection and modifies the color of the strokes.
foreach (ContextLink contextLink in
contextNode.Links)
{
contextLink.SourceNode.Strokes.
ModifyDrawingAttributes(new
DrawingAttributes(Color.Blue));
contextLink.DestinationNode.Strokes.
ModifyDrawingAttributes(new
DrawingAttributes(Color.Red));
}
The sample program above uses blue highlights for
SourceNodes and red highlights for DestinationNodes. Listing 3 shows the complete contents of
the FindLinkedNodes methods.
The ability to determine relationships between
nodes can add powerful functionality to any inkaware application. A good example would be in ed-
Figure 7: The InkAnalyzer knows that the underline
underneath the word “this” is related.
Listing 3: FindLinkedNodes method
private void FindLinkedNodes(ContextNode contextNode)
{
if (contextNode.Links.Count > 0)
{
foreach (ContextLink contextLink in contextNode.Links)
{
contextLink.SourceNode.Strokes.ModifyDrawingAttributes(new
DrawingAttributes(Color.Blue));
contextLink.DestinationNode.Strokes.ModifyDrawingAttributes(new
DrawingAttributes(Color.Red));
76
}
}
// Recursively Parse Tree
foreach (ContextNode subNode in contextNode.SubNodes)
{
FindLinkedNodes(subNode);
}
}
Microsoft.Ink.RootNode [Ink Analysis can 1. Recognize text 2. Understand Structure 3. Recognize Shapes]
Microsoft.Ink.WritingRegionNode [Ink Analysis can 1. Recognize text 2. Understand Structure 3. Recognize Shapes]
Microsoft.Ink.ParagraphNode [Ink Analysis can]
Microsoft.Ink.LineNode [Ink Analysis can]
Microsoft.Ink.InkWordNode [Ink]
Microsoft.Ink.InkWordNode [Analysis]
Microsoft.Ink.InkWordNode [can]
Microsoft.Ink.ParagraphNode [1. Recognize text]
Microsoft.Ink.InkBulletNode [1.]
Microsoft.Ink.LineNode [Recognize text]
Microsoft.Ink.InkWordNode [Recognize]
Microsoft.Ink.InkWordNode [text]
Microsoft.Ink.ParagraphNode [2. Understand Structure]
Microsoft.Ink.LineNode [Understand Structure]
Microsoft.Ink.InkWordNode [Understand]
Microsoft.Ink.InkWordNode [Structure]
Microsoft.Ink.ParagraphNode [3. Recognize Shapes]
Microsoft.Ink.LineNode [Recognize Shapes]
Microsoft.Ink.InkWordNode [Recognize]
Microsoft.Ink.InkWordNode [Shapes]
Table 1: Structure of Figure 3.
iting scenarios. Users mark up some ink, an image,
or text with ink annotations and the annotations
follow their targeted content. Another example
would be in diagram drawing scenarios, where the
lines between nodes hold intrinsic data about the
relationship between nodes.
Event-based Model
The InkAnalyzer triggers over a dozen events in
real time as it performs its analysis. This eventbased model is useful for cases when integrating ink
analysis into existing code, such as inserting ink directly into a legacy binary file format.
For instance, you can insert the data retrieved during analysis in real time into an existing file format.
Another example would be looking for a certain
word or phrase that could open a window, create a
new file, or trigger any kind of event.
Background Analysis
in both the user interface and the ability to add any
more ink. Remember that in addition to analyzing
ink, your applications may already be working on
other tasks.
Background analysis will not stop the user interface
from responding. It runs the analysis on a background thread and will send an event when it has
concluded.
If that weren’t reason enough to analyze ink in the
background, there are more benefits. If the user
adds more ink during the analysis, the InkAnalyzer
will automatically combine the new results into the
tree while keeping track of the new strokes, which
haven’t been analyzed yet. The user can be blissfully unaware of this and does not need to merge
results.
Ellipse
Circle
Triangle
IsoscelesTriangle
EquilateralTriangle
RightTriangle
Incremental Analysis
The InkAnalyzer also supports incremental analysis, which avoids reanalyzing ink by only analyzing
new content added since the last analysis.
If I had ten pages’ worth of ink to analyze and my
code ran the analysis on a foreground thread, my
user interface would lock up until the analysis completed. For anything but simple applications, this
amount of lag time would be unacceptable.
For example, consider appending a new sentence
to a large amount of ink. Rather than analyze the
entire amount again, the InkAnalyzer will only analyze the new content.
However, as the amount of ink to analyze grows, so
does the likelihood of users experiencing lag times
It does, however, examine nearby areas to check
if the new ink changes any of the previous results.
None
Quadrilateral
Rectangle
Square
Diamond
Trapezoid
Parallelogram
Pentagon
Hexagon
Table 2: Shapes returned from the
InkAnalyzer
77
For example, adding an underline to a word in the
middle of ink.
Practical Applications of Ink Analysis
Ink analysis has a number of practical applications.
The ability to interpret structure as well as content
from written text opens up a number of possibilities.
Here are some ideas that come to mind.
The Ink to Blog Application
One practical use of the ink analysis engine would
be creating a Tablet PC-friendly, blog-writing application. Blog posts generally have a document structure of title and content. Using
the hint area attribute, you can
easily isolate the blog post title
ability to interpret
from the main document content.
as well as content from
The
structure
written text opens up
a number of possibilities.
You can use the InkAnalyzer to
analyze the document further
to pull out bulleted lists, paragraphs, etc.
With a program like this, writing posts to a weblog
could literally become as easy as writing into a journal.
The Ink to SharePoint Connector
Of course, this type of application need not be limited to weblogs. Enterprises would benefit greatly
from a connector application for Tablet PC and Microsoft SharePoint®.
As SharePoint use increases, so does the value of
a quick and easy way to input ink data into SharePoint. A good scenario would be electronic forms,
perhaps even forms filled out by mobile workers.
Rather than analyze
the entire amount again,
the ink analyzer will
only analyze the new content.
You could improve the engine by adding XSLT
support to automatically transform the basic node
types into an XML document that would conform
to a specific structure.
Recognition Game
Another potential use for the ink analysis engine
would be both fun and educational. The program
would tell the user to draw a particular shape and
examine the ink to determine if the shape was
drawn.
Conclusion
The ability to write notes and convert the ink to text
is one of the Tablet PC platform’s most remarkable
features. However, to really leverage the power of the
platform, handwriting recognition is not enough.
Users expect their Tablet PCs to understand not just
their writing, but also their notes, their annotations,
and their drawings and diagrams.
The InkAnalysis API provides a rich set of functionality that Tablet PC and mobile PC developers can
use to deliver applications that meet these expectations.
Frank La Vigne
This application has a similar architecture to the
ink-to-blog application. You could define areas in
the main writing area to correspond with any given
SharePoint form. From there, you could use the ink
analysis engine to separate the fields and interact
directly with the SharePoint server via a Web service. Ink one minute, data entered into SharePoint
the next.
Imagine the potential for field workers and sales
personnel to instantly convert and submit handwritten notes and documents into SharePoint server.
The Ink-to-XML Engine
You could power the two previously mentioned
examples by one, general-purpose ink-to-XML engine. This engine would take the tree structure defined by the InkAnalyzer and convert the nodes to
corresponding nodes in an XML tree.
78
Best Practices in Game
Design for the Ultra-Mobile PC
The Ultra-Mobile PC (UMPC) expands the market for PC games
that run on Microsoft® Windows® XP, as long as developers
consider certain design requirements to ensure a good user
experience. In most cases, a single version of games can span both the UMPC
and traditional PC platforms.
Matt Gillespie
T
he Ultra-Mobile PC (UMPC) platform is of
Screen-Size Considerations
increasing importance as a target system for
game developers. Because these systems run
Because the UMPC screen is much smaller than
on Microsoft Windows XP Tablet PC Edition,
a traditional screen, the size of graphical elements
they don’t require a full opermust be handled with some
ating system port for existing
care. Developers must avoid
Fast Facts
games, but the UMPC form
making scaled-down graphical
Since Windows-based
factor does introduce a numelements too small, as well as
UMPCs run a full version
ber of unique considerations
allowing elements that are left
of the operating system,
in game design. For example,
the same physical size to conUMPCs typically have a touch
sume too much screen space.
providing UMPC
screen with an 800x480 or
Specifically, text and icons are
support for games is typically
1024x600 resolution, which
often hard to see clearly, some
relatively straightforward and
are small pixel sizes and unbuttons or units are difficult to
very cost effective.
usual aspect ratios. Developclick reliably, and some game
The best practices in this
ers must also accommodate
windows do not fit entirely
article demonstrate to game
alternatives to conventional
onto the screen.
developers how to resolve
keyboard and mouse user interaction, and a CD-ROM
some of the primary issues
drive is typically not availText and Icon Sizes
involved in successfully
able. See the sidebar, Typical
porting games
UMPC Hardware.
Issue: In the interfaces of
to the UMPC platform.
games that are ported to the
The best practices in this docUMPC, text or icons may
ument for enabling PC games to run successfully
shrink to a size where it becomes difficult to see
on the UMPC platform are based on an analysis
them clearly. This prevalent issue is important to
of a large number of PC games currently on the
avoid, because text and icons become useless if
market. Those games were executed on a UMPC
they cannot be read or differentiated. Text that
to identify common strengths and weaknesses
appears reasonably sized on a 15” screen can
associated with the platform, as well as specific
easily become too small when shrunk to a fivegame-design factors that contribute to the best
to seven-inch screen that you might find on a
user experience. The analysis includes design
UMPC. Aside from the actual font size of text,
considerations for providing high-quality games
chat and other text windows may become too
on the UMPC, as well as common issues assosmall. Accommodating smaller window sizes by
ciated with providing UMPC support, providing
decreasing the font size can make the text difbest practices to resolve each of those issues.
ficult to read.
Ideally, games designed with
the UMPC in mind should
use text sparingly
and consider the UMPC screen
size when choosing a font.
Best Practice: Ideally, games designed with the
UMPC in mind should use text sparingly and
consider the UMPC screen size when choosing
a font. Likewise, icons should not rely heavily
on fine details, so different objects are different enough to be easily distinguished from one
another, even on the smaller UMPC screen. In
places where increasing text size enough would
compromise other elements of the game, allowing text size to be adjustable may be a good solution, allowing individual players to decide for
themselves the ideal size for the text.
Matt Gillespie is an independent
technical author and editor
working out of the Chicago area
and specializing in emerging
hardware and software
technologies. Before going
into business for himself, Matt
developed training for software
developers at Intel Corporation
and worked in Internet Technical
Services at California Federal
Bank. He spent his early years
as a writer and editor in the
fields of financial publishing and
neuroscience.
Michael Finkel
Michael Finkel is a Technical
Marketing Engineer in the
Ultra Mobile Group at Intel®
Corporation. Michael has been
with Intel since 1998 and has
worked on graphics software
and optimizing performance for a
variety of platforms ranging from
handhelds up to servers. Michael
is now focused on the definition
and implementation of Ultra
Mobile PC platforms.
Best Practices in Game Design for the Ultra-Mobile PC
79
Victoria Bailey
Victoria Bailey is a Senior in
Computer Science and German
at Oregon State University. In
Summer of 2006 she took part in
an internship at Intel in Folsom,
California, where she wrote this
article. In the future, she plans
to attend graduate school for
Computer Science.
This article also appears on
www.intel.com/software/
mobile and is reprinted
with permission from Intel
Corporation.
Best Practice: The key to resolving this issue is for game
developers to consider the
800x480 and 1024x600 resolutions as they develop game
interfaces for the UMPC,
shrinking the entire game
to fit on the screen or rearranging the interface to take
full advantage of the wide
screen area. Certain simple
accommodations can make
games more playable, even
in a truncated state. For exFigure 1. On a 20-inch PC monitor (left), the user can easily identify and select a
ample, providing scrollbars
domino using the arrow cursor. When the domino images are scaled to fit on a
and allowing the window
five-inch UMPC screen (right), however, it becomes very difficult for users to select
to be resized (manually or
individual dominoes, or even to identify how many dots are on each domino.
automatically) may be sufficient in some cases to allow
Clickability of Buttons and Other
the user access to all parts of the screen and to provide an acceptable user experience, particularly for
Elements
browser-based games.
Issue: Similar to the issue of small text, buttons
and other clickable game elements add complexity
to porting games to the UMPC. While the overall interface on the UMPC must be smaller than
the corresponding standard PC version, the actual
buttons in the UMPC interface must be larger, in
order to accommodate being accurately clicked
by a stylus or finger, rather than the more precise mouse that is used in the standard PC version. This issue is particularly acute when multiple
buttons or other elements are clustered together,
which makes it more likely that the user will select
a different element than the one they intend. Even
when it is possible to select the correct element
with some care, this issue can significantly detract
from the user experience.
Best Practice: As with the size of text and icons,
game developers should avoid this problem by using large, distinct buttons and other elements. These
elements should also have enough space between
them, making it less likely that the user will inadvertently select the wrong one. In those cases where
a text label appears adjacent to a button or other
element, that label should be part of the clickable
area associated with the element, making it easier
to click without having to devote any additional
space.
Game Window Size
Issue: If game windows use a fixed size, rather
than automatically resizing themselves to fill the
full UMPC screen, parts of the game may not be
visible all at once, and, in some cases, certain parts
may not be reachable at all, as shown in Figure 1.
This situation is made more complex by the potential for players to use the 800x480 or 1024x600
resolution enabled by the UMPC’s wide screen,
since those aspect ratios must be matched to the
standard 4:3 aspect ratio employed by most PC
games.
80
Touch Screen Considerations
The use of a touch screen instead of a mouse on
the UMPC adds another layer of complexity to
porting games. For most software purposes, the
touch screen behaves like a mouse, except that instead of registering a linear track of movement as
the user drags the cursor, it generates a series of
button-down events with a location each time the
user touches the screen. The inability to move the
cursor without clicking and the relationship between left-clicking and right-clicking also causes
issues.
Accurate Interpretation of Tap-Only
Touch Screen Input
Issue: UMPC games must be able to interpret cursor input provided by the user as a series of points
(corresponding to a series of touch-screen taps),
rather than requiring a linear pattern of movement
(like that provided by mouse movement controlling
the cursor). This issue tends to arise particularly often, for example, in games that shift the user’s perspective so that the cursor is always at the center of
the screen (which is common in first-person shooters). In such a scenario, when the player touches
the screen, the cursor may jump around the screen
erratically, instead of moving to where the user
touched.
Best Practice: If the game tracks movement of
the cursor from point A to point B, it must be
able to interpret a click at point A, followed by
a click at point B, without the cursor first moving between point A and point B. Requiring an
external mouse to be plugged into the UMPC will
typically correct this problem, although that solution is clearly sub-optimal from a user-experience
point of view.
Figure 2. On a standard PC (left), the user is able to see the entire play area, including informational parts of the player
interface. If the game window is cropped to fit the UMPC’s smaller size and different aspect ratio (center), part of the game
will be hidden, represented here by the translucent parts of the screen at the edges. If the game window is scaled to fit the
UMPC screen (right), all parts of the game can readily be made visible, with a level of visual distortion that may be acceptable.
Accurate Touch-Screen Mapping
The touch screen can be mapped incorrectly to the
game when the game runs in full-screen mode but
does not take up exactly the full screen (either not
filling the entire screen or taking up more than the
full screen).
Issue (Game Panel Smaller than Screen): When
the desktop is centered with space on either side
(as in the case of a 640x480 panel being centered
on a 800x480 screen or an 800x600 panel being
centered on a 1024x600 screen, with black bars
on each side), mismapping may occur between the
two, with the entire touch-screen surface being
mapped to the relatively small display area. That
mismapping, as shown in Figure 2, results in inappropriate response of the interface to user interaction, such as a button’s clickable screen area being
located away from the visual representation of the
button.
Best Practice: Games can be centered with black
bars at the sides without causing the touch screen to
be mismapped, as long as the mapping logic considers the black bars as part of the screen (even though
they are not used as a part of the game). With this
adjustment, touches on the screen map to exactly
the location touched.
Issue (Game Panel Larger than Screen): In those
cases where the game window takes up more than
the full vertical space of the screen, the touch screen
may be incorrectly mapped vertically instead of horizontally, as shown in Figure 3. This scenario typically causes the touch screen to be mapped to the
whole game window, rather than just to the visible
section. In such cases, clicks do not correspond to
the area of the game window that the user intends.
Best Practice: Developers should ensure that the
operating system maps the touch screen exactly to
the entire visible portion of the screen (including
portions not being used by the game), and not including portions of the game window that are not
visible. Another solution is for developers to natively support the 800x480 and 1024x600 screen
resolutions as a user-configurable option. In some
cases, it might be possible for the game to automatically stretch the window to fit the full screen, if the
resulting dimensional distortion of game elements
is acceptable.
Mark Hopkins
Lead Writer, Mobile & Tailored
Platforms Division
The UMPC is the perfect mobile
solution. It’s small enough to take
almost anywhere, yet powerful
enough to run full Windows Vista.
Alternatives to Hover-Over Effects
Figure 3. If the game window is centered with black bars
on each side, the game logic that interprets screen taps
must avoid mapping the entire physical screen area to the
smaller game window. Such incorrect mapping can cause
a screen tap at one position (represented by the yellow
arrow) to be interpreted by the game as occurring at a
different position (represented by the red burst).
Issue: Many games use a hover-over feature that
enables the user to position the mouse over an object or location in the game (without clicking on
it) to trigger an informative animation or tooltip.
This feature, which is often used to provide instructions for novice players, is incompatible with touch
screens, since a touch screen cannot move the cursor without clicking.
Best Practice: Since the hover-over feature is not
compatible with current UMPC hardware, develop-
81
Additional Resources
Form Factor Considerations
Game developers and architects
who are considering how best to
integrate the needs of the UMPC
platform into their offerings
will benefit from the following
resources:
• Intel® Software Network
Mobile Developer
Community (http://
softwarecommunity.intel.
com/isn/home/Mobility.
aspx) is a hub for developer
information related to all things
mobile, including technical
documentation, SDKs, forums,
knowledge bases, and blogs.
• Intel’s UMPC Platform site
(http://www.intel.com/
design/mobile/platform/
umpc.htm) introduces the
platform and the possibilities
it engenders for the developer
and user communities.
• The official Microsoft UMPC
Platform site shows latest
information about UMPC
hardware and software.
(http://www.microsoft.com/
umpc)
• UMPC Community (http://
umpc.com/default.aspx)
provides information on
devices, applications, and
usage models, as well as
hosting discussion forums for
users and developers.
• UMPC Buzz (http://www.
umpcbuzz.com/) links to
news items, blogs, forums,
and software downloads for
the UMPC.
A number of issues relate directly to hardware
differences in the UMPC form factor, relative to
standard PCs. For example, unlike a standard
PC, a UMPC may have a touch screen, joystick,
user-programmable buttons, and dedicated buttons such as a menu button, but it will typically
not have a keyboard or CD-ROM drive. Although
it may be possible to add a keyboard, CD-ROM
drive, or other peripheral devices, doing so reduces
the portability of the UMPC. Moreover, different
models of UMPCs may have different elements
included in their designs, which causes the limitations associated with the form factor to vary by
device. That variability typically requires developers to focus on the lowest common denominator of
hardware features when designing games, in terms
of core requirements.
Figure 4. Having part of the game window (represented
by the translucent area) cropped from the physical
screen may also result in a screen tap at one position
(represented by the yellow arrow) being interpreted by the
game as occurring at a different position (represented by
the red burst), due to mismapping between the different
shapes of the game window and the physical display.
ers should choose alternative events to trigger the animation or tooltip. For example, it could be triggered
when the player reaches the part of gameplay when
the subject of the animation or tooltip becomes relevant. Another possibility is to design the interface
with a means of temporarily interpreting click events
as mouseover events, such as a button that can be
held down to allow touches on the touch screen to
move the cursor without registering a click.
Making games
able to be downloaded and
installed via the Internet
is an ideal alternative to CDs
for UMPC games.
Accommodating Right-Click
Functionality
Issue: On a typical UMPC touch-screen, users perform right-clicks by holding down the click on the
screen for a longer time than for a left-click, which
can cause users to inadvertently left-click when they
mean to right-click. Moreover, it is impossible to
perform a left-click and right-click simultaneously.
Best Practice: Developers should provide alternative user interactions to replace right-click functionality, such as using double-clicks or clickable controls on the screen that take place instead of rightclicking. Alternatively, game developers can simply
require the use of an external mouse or other pointing device, although that requirement may detract
from the user experience.
82
CD-ROM Drive
Issue: CDs are currently the most popular method
for distributing games. In addition to being installed from a CD, the disk is often required as
a means of verifying ownership of the game, as
an anti-theft measure, or to dynamically load content from the CD as it is needed, such as video
sequence. Because UMPCs typically do not have
integrated CD-ROM drives, this modality requires
the use of an external drive, which hampers the
portability of the UMPC. Further, users may not
even own an external CD drive. Notably, this issue has no impact on web-hosted, browser-based
games.
Best Practice: Making games able to be downloaded and installed via the Internet is an ideal
alternative to CDs for UMPC games. This distribution model is already in widespread use, and
developers should consider the emergence of
UMPCs as a further impetus to expand upon it.
They should also consider using certificates or
other software-based means to verify ownership,
rather than requiring the physical presence of the
CD. Rather than dynamically loading video and
other content from the CD as it is needed during
gameplay, developers should consider providing
the option for that content to be hosted on the
UMPC’s hard drive.
Developers should
provide alternative user
interactions to replace rightclick functionality, such as using
double-clicks or clickable
controls on the screen that take
place of right-clicking.
Limiting the Need for Diverse
Command Inputs
Issue: The UMPC form factor provides for a
limited number of inputs, relative to the full keyboard and mouse available from a standard PC.
Even those games that are designed to be run on
a PC using a joystick may require the use of multiple buttons that are included on a typical joystick
controller, which may not be available from the
UMPC. UMPCs also have issues with games that
incorporate keyboard shortcuts as a key gameplay component. Notably, games that are primarily mouse-driven are not affected by form-factor
limitations in this area.
Best Practice: Developers can address limited input options on the UMPC by decreasing the number of commands that are actually
required to play the game, as opposed to being optional conveniences such as shortcuts for
menu options. A somewhat more flexible solution, however, is to create buttons on the screen.
Particularly if the buttons are context-sensitive,
appearing only when they are relevant, this can
provide an open-ended solution for input requirements.
Conclusion
Game developers and architects providing support for the UMPC platform in mainstream game
offerings can develop a discrete set of design considerations that address common issues that arise
during the development process. The best practices
described in this document provide a foundation for
that effort. Because Windows-based UMPCs run a
full version of the operating system, it typically requires less effort to provide this support than a full
port to a separate operating environment. Thus, an
incremental effort may expand a game’s market to
include the expanding deployment of the UMPC
platform, providing a competitive advantage to the
game software provider.
Matt Gillespie
Michael Finkel
Victoria Bailey
Typical
UMPC Hardware
A typical UMPC consists of (see
Samsung Q1 or Asus R2H for
example):
• <=1GHz CPU
• 512MB-1GB RAM
• 40-60GB HDD or 20-30GB SSD
• 5-7” LCD Screen
• Touch screen
• 2-6 hardware buttons
• Cursor and/or Mouse control
• Wireless
• Bluetooth
• USB
• Built-in keyboard (might be
present)
Optional devices may be present
when not mobile:
• Add-on keyboard (BT or USB)
• Add-on mouse (BT or USB)
• External HDD
• External CD/DVD
Addressing Keyboard Requirements
Issue: In addition to issuing commands, many
games also use the keyboard for input such as
naming characters, creating profiles, saving
games, or supporting a chat feature for online
games. It is common for games to require a keyboard, for example, to enter a profile name at the
beginning of a gaming session but not to require
the use of a keyboard at any other time during
the session. Some games also require players to
use the keyboard to name their saved games. In
many cases, the on-screen keyboard will not run
on top of the game interface, so it cannot resolve
these issues.
Best Practice: Developers should either design
games explicitly not to conflict with the onscreen
keyboard (e.g., by not defaulting to full-screen
mode) or provide an on-screen keyboard within
the interface itself that appears only when needed. One simple means of minimizing the need for
keyboard input is to provide default values for
text strings such as profile names, saved game
names, etc. and to allow them to be selected using screen-tap input. Additional values for these
text strings can also be provided by adding an
interface button that randomly selects a new value from a list of character names created by the
developer. Saved games can also be identified using a screen capture of where the player left off,
together with a date and time stamp. Since chat
features are very rarely a core aspect of gameplay, and they can generally be disabled on the
UMPC without negatively impacting the user experience.
83
SideShow Development in Depth
Providing access to information on your PC without having to
boot the PC is one of the goals of Windows SideShow™.
Windows SideShow device technology consists of a separate screen, CPU, and
memory that you can use to view this information. The devices range from displays on
the back of laptop lids to remote controls to credit-card sized screens you can put in
your pocket. This article will explain how your applications can provide information
on a Windows SideShow device.
Dr. Neil Roodyn
[email protected]
Dr. Neil is an independent
itinerant consultant, trainer,
and author. He travels the
world working with software
companies. Neil brings his
business and technical skills to
the companies he works with to
ensure he has happy customers.
Dr. Neil worked with the
Windows SideShow team prior
to the release of Windows Vista
to help increase the value of the
developer APIs and samples.
O
riginally termed auxiliary display technology,
and some will be 600 x 400 pixel color displays. A
Windows SideShow is a Windows Vista™ opWindows SideShow device is not guaranteed to have
erating system technology. Microsoft ships the
any specific capabilities. The code that you will write
software components needed
to display your information on
to utilize a Windows SideShow
a SideShow device will create a
Fast Facts
device as part of Windows Vista.
gadget and encode the data for
Each application that wishes to
a specific end point. The advanWindows SideShow gadgets
display content on a Windows
tage of providing these defined
are applications
SideShow display needs to creformats is that you do not need
that run on the PC and send
ate a Windows SideShow gadto be concerned about writing
content to the Windows
get. The term gadget can someany device-specific code. All of
SideShow device.
times be confusing as there are
your code will run on the PC
other gadget technologies that
and send the content to the deEach SideShow gadget has
are not directly related to Winvice.
a unique identifier (GUID)
dows SideShow.
known as the Application ID.
The end points supported on
The different end points are
This article will explore the
Windows Vista are SCF (Simple
also identified by a GUID,
different options you have to
Content Format) and iCalendar.
or Endpoint ID.
add your application-specific
SCF is a new XML data format
Each piece of content for a
content to Windows Sidethat you can use to describe
Show supported devices. The
text, images, dialog boxes, and
Windows SideShow gadget
samples that ship with the Mimenus, as well as navigational
has a unique identifier called
crosoft® Windows® SDK are
constructs. All Windows Sidethe content ID.
all written in C++ and utilize
Show compatible devices must
the COM APIs Windows Sidesupport SCF. The iCalendar
Show supports. I’ll start the exploration here.
format provides a consistent way to provide calendaring content to the device. However, not all deThe majority of this article will focus on the Manvices will support the iCalendar format.
aged .NET Framework API and how you can use it
to rapidly create Windows SideShow gadgets. You
Windows SideShow manufacturers can define their
will learn how to add different types of content for a
own end points. These custom data formats allow
gadget to display and how you can define the navithe OEMs to provide extended feature sets that
gation between that content.
your applications can reach.
You can also create a Windows SideShow gadget
through another Microsoft gadget technology: Windows Sidebar. Windows Sidebar provides a simple
API for creating a Windows SideShow gadget and
adding content from a Sidebar gadget.
How it Works
Windows SideShow devices all support a well-defined set of end points. An end point defines a certain
format of content that the Windows SideShow devices can display. How the device displays the content is
not defined. A wide range of devices with varying capabilities will ship in the coming year, some devices
will have monochrome screens, some will have color
screens, some screens will be 2-line LCD displays,
84
A single Windows Vista PC can support multiple
Windows SideShow devices. Users will want to
have different information displayed on the different devices. In the Windows Vista Control Panel
is a Windows SideShow Control Panel. Users can
use this SideShow Control Panel to manage which
SideShow gadgets will run on which devices. Figure 1 shows an example of the Windows SideShow
Control Panel.
If you are not lucky enough to have a real Windows
SideShow device, the Windows Platform SDK ships
with a device simulator (Figure 2). You can find this
simulator in the \bin folder of the Windows SDK.
The simulator executable is called VirtualSideShow.
exe, but before you run the simulator you will need
to register it on your machine by running the Win-
Greg Parks
Development Manager
& Architect, Windows
SideShow
We’ve just scratched the
surface of what we can do
with the Windows SideShow
platform. The more you think
about the possibilities, the more
revolutionary and exciting those
possibilities become.
Figure 1: Windows SideShow Control Panel.
dowsSideShowVirtualDevice with the /regserver
parameter. Note that you will need to run the command with elevated permissions for the registration
to take effect.
Microsoft originally designed for Windows SideShow devices (and early prototypes) to run using
the .NET Micro Framework. It has become clear
that a range of devices available will not run the
.NET Micro Framework. As already stated, a Windows SideShow device simply needs to support the
required end points. At a number of conferences
in 2006, vendors demonstrated Windows Mobile™
devices with support for the SCF end point. If you
want to reach the maximum number of Windows
SideShow devices, the best option is to send SCF
content to the Windows SideShow APIs and leave
the rest to the Windows SideShow platform.
Glance content is the summary information the
user will see next to the Gadget icon before they
activate the gadget. Typically this will be one or
two lines of text. Microsoft recommends 2-12
lines and varies by gadget. Glance content always
has a content ID of 0 and is UTF-8 encoded text.
Nothing else can use this content ID. The Glance
content is simply text, you should not provide it in
XML tags. Other end points will also use a content
ID of 0 for their Glance text and again this will be
a plain text field.
All the other SCF content types require the content
to be provided within a body element. The body
element specifies the start and end of the content.
Each Windows SideShow gadget has a unique identifier (GUID); this is known as the Application ID.
SideShow also identifies the end points by a GUID,
or Endpoint ID. Each piece of content for a SideShow gadget has a unique identifier (a 32-bit unsigned integer) called the Content ID.
Simple Content Format
SCF (Simple Content Format) is an XML-based
format that defines content to be rendered to a device. With SCF, an application can send four different types of content to the device: Glance Content,
Menus, Content Pages, and Dialog Pages.
Figure 2: Windows SideShow Simulator.
85
Dan Polivy
Program Manager,
Windows SideShow
Windows SideShow is one of
the new and exciting features in
Windows Vista that will change
the way people interact with
their PCs and devices. It’s been
an incredible opportunity to
build this platform from some
simple ideas to what it is today.
It has been our goal to allow
our partners to innovate on the
platform, and I’m just amazed at
the breadth and type of devices
they’ve built so far. I can’t begin
to fathom what they’ll do in the
future.
Consider the body tag as the root node for all of
the SCF XML.
A menu is a useful component for navigation between content in the gadget. A menu is essentially a
list of items, with each item linking to other content
IDs. These content IDs could be for other menu
pages, content pages, or dialog pages. Windows
SideShow has the concept of a context menu and
many devices will have a hardware button to summon the context menu. A menu must have a content ID and can optionally have a title and modify
the action to take when a user selects a menu item.
You should have a title on a menu page to inform
the user what they are selecting. The default action
taken when a user selects a menu item is to navigate to the page with the content ID identified by
the menu item’s target attribute. By changing the
selectaction attribute of a menu, when users select
a menu item they will navigate to the context menu
specified by that menu item’s menuid attribute. A
menu for selecting content pages with images might
look like this in SCF:
<body>
<menu id="1" title="Image List">
<item target="2">Sunset</item>
<item target="3">Winter</item>
<item target="4">Water lilies</item>
<item target="5">Blue hills</item>
</menu>
</body>
The only required attribute for a menu item is the
target. You can include up to five other optional attributes to uniquely identify menu items, such as:
add images to display as icons next to a menu item,
set a menu item as the default, enable or disable
menu items, and identify context menus for a menu
item.
Content pages will most likely represent the main
part of your gadget information. Content pages can
contain text, images, and buttons—the buttons define the hardware button actions and do not get
displayed on the page. You use the content element
to define a content page. You can style and align
each of these items of content. The only required
attribute for a content element is the id, but again
using the title attribute is recommended. A content
page can also have a background image and define
how you can lay out the background image, using
the bg and bgfit attributes. You can define a menu
as the context menu for a content page by using the
menuid attribute of the content element. The code
snippet below shows the SCF for a simple content
page with some test and an image:
<body>
<content id="2" title="CoDe Focus">
<txt align="c">
CoDe Magazine rocks!
</txt>
<txt>Coding is fun</txt>
<img id="100"
alt="[CoDe.jpg]"
86
fit="screen" />
</content>
</body>
The dialog page content allows a gadget to show a
simple dialog box with an icon, title, message, and
buttons. Use the dialog element in SCF to define
dialog pages and specify the content ID using the
id attribute. The optional title attribute will inform
the user of the purpose of this dialog box. Using
the imgid attribute to identify the content ID of an
image will inform the Windows SideShow device
to display that image in the dialog page. Similar to
the content element, a dialog element can contain
txt and btn elements. A dialog page cannot contain
img or br elements though. In the example below,
you should note that it is possible to add br elements to the txt elements that are children of the
dialog element:
<body>
<dialog id="4" title="SideShow Dialog">
<txt align="c">
<br/>This is a SideShow Dialog.
<br/><br/>Do you like it?
</txt>
<btn key="Enter" target="10">Yes</btn>
<btn key="Back" target="11">No</btn>
<btn key="Left" target="12">Not Sure</btn>
</dialog>
</body>
COM Interfaces
The simplest way to build a Windows SideShow
gadget using the COM API is to create a C++ executable that CoCreates a SideShowSession object.
The ISideShowSession interface provides methods
that allow an application to register for sending
content and notifications to the SideShow devices
displaying the gadget content. When an application
calls the RegisterContent method it passes three
parameters: a GUID identifying the gadget, an EndPoint ID, and a memory address for the platform
to fill with a pointer to an object that supports the
ISideShowContentManager interface:
::CoCreateInstance(CLSID_SideShowSession,
NULL,
CLSCTX_INPROC_SERVER,
IID_ISideShowSession,
(LPVOID*)&m_pSession);
if (SUCCEEDED(hr))
{
hr = m_pSession->RegisterContent(
HELLOWORLD_APPLICATION_ID,
SIDESHOW_ENDPOINT_SIMPLE_CONTENT_FORMAT,
&m_pContentMgr);
.
.
.
The GUID (HELLOWORLD_APPLICATION_ID)
is the Application ID of the gadget discussed pre-
viously. In order for the Windows SideShow platform to know about this gadget, you need to add
this GUID to the registry in the key: HKCU\SOFTWARE\Microsoft\SideShow\Gadgets\ for per-user
gadgets and: HKLM\SOFTWARE\Microsoft\SideShow\Gadgets\ for gadgets that are available to all
users of the machine. Your gadget should add the
GUID as part of the install program.
Once you have a pointer to an object that implements the ISideShowContentManager interface,
use the Add method to add content to the Windows
SideShow device This Add method takes one parameter: a pointer to an object that implements the
ISideShowContent interface. This is an object that
your gadget code needs to create. The ISideShowContent interface contains three methods (in addition to the three IUnknown methods): GetContent, get_ContentId, and get_DifferentiateContent
(these are properties but implemented in COM as
methods).
For every piece of content that a gadget wishes to
display on the device, you will need to create an
object that implements the ISideShowContent interface. The GetContent method is called from the
Windows SideShow platform to retrieve the content to display for the content ID represented by
this object:
STDMETHODIMP CHelloWorldContent::GetContent(
ISideShowCapabilities *pICapabilities,
DWORD *pdwSize,
BYTE **ppbData
)
{
HRESULT hr = S_OK;
char* szXML = "<body><content id=\"1\"
title=\"Windows SideShow Hello World\"><txt
align=\"c\" wrap=\"0\">Hello Windows SideShow
World</txt></content></body>";
*pdwSize = (DWORD)strlen(szXML) + 1;
*ppbData = (BYTE*)::CoTaskMemAlloc(*pdwSize);
if (NULL != *ppbData) {
hr = StringCchCopyA(
(LPSTR)*ppbData, *pdwSize, szXML);
}
else{
hr = E_OUTOFMEMORY;
}
return hr;
}
The code snippet above shows how to send XML
content to the device. Microsoft anticipates that the
gadget will generate or load the XML rather than
be hard coded. This example shows how simple it
is to send SCF pages to the device. Images present a slightly more complex issue. An object that
represents image content for the device must still
implement the ISideShowContent interface. In the
GetContent method, the image being sent to the
device must be placed in the byte array ppbData.
Many devices will have limited resources, so a gad-
get must first determine the capabilities of the device, and then convert the images into a compatible
format. The platform implements the ISideShowCapabilites interface in an object reference passed
into the GetContent method.
Building a Windows SideShow gadget using the
COM API might be the most challenging way to
place content on the device, but it is also the most
powerful. The COM API contains the full feature
set provided by Microsoft to interact with the Windows SideShow platform.
Building Managed SideShow Gadgets
The COM API is fairly straightforward and would
not be overly hard to wrap in a managed library.
Fortunately Microsoft has done this already. By the
time this article goes to print, Microsoft should have
released the Windows SideShow Managed API. I’ll
base what now follows in this article on the existing
MSDN documentation and a prerelease version of
the Managed API.
In order to help explain how the Managed API
works, you can download the sample application
from http://Roodyn.com/SideShowContentCreator.ashx
The sample application allows the user to add a different type of page to a Windows SideShow device.
You can see the application running alongside the
simulator in Figure 3. Each of the tabbed pages enables the user to add a type of content to the gadget.
Before a user can add any content, the user needs
to register the gadget. The entry in the registry allows the Windows SideShow platform to find the
gadget and provide the user an option to add it to
their device.
In order to register the gadget it must have a GUID
known as the Application ID. At the top of the
Mainform.cs file, declare a static member variable
of type Guid:
private static readonly Guid s_gadgetId =
new Guid(
"{D1AC24B8-8349-4816-9270-5050C3D9009A}");
You can then use the static
method Register on the static
class
Microsoft.SideShow.
GadgetRegistration to register
the gadget for Windows SideShow use. This Register method takes parameters to identify
the:
The COM API contains
the full feature set provided by
Microsoft to interact
with the Windows SideShow
platform.
• Registry key to use (local
machine or current user)
• The gadget identifier
• The end-point to use
• A name for the gadget (that will appear in the
Control Panel and next to the Gadget icon on
the device)
87
• A command to use when starting the gadget
• The icon to use to display the gadget
• Online or offline usage
• The cache policy
• A GUID for a property page for the gadget.
the Glance content. Updating the Glance content
is something that most applications should do on
a regular basis to provide hints to the user as to
changes in the content of the gadget:
_gadget.AddGlanceContent(glanceTextBox.Text);
This covers everything a Windows SideShow gadget can put in the registry. In the MainForm_Load
method of the sample, the call is made to register
the gadget. In production code you may prefer to do
this in the installer:
string iconPath =
Environment.CurrentDirectory + "//icon.ico";
GadgetRegistration.Register(false, s_gadgetId,
ScfSideShowGadget.ScfEndpointId,
"Dr. Neil's SideShow Gadget", null,
iconPath, false,
GadgetCachePolicies.KeepNewest, null);
As this gadget will supply content in SCF, a member variable of the MainForm class represents the
gadget:
private ScfSideShowGadget _gadget;
The _gadget object is instantiated in the MainForm_Load method. The ScfSideShowGadget
class is designed to help you write gadgets that
send SCF to the device. Take note that the GUID
of the gadget is required to create the object. This
allows the Managed API to connect the gadget to
its registered properties and call the methods in
the native code to manipulate the correct gadget
on the device:
_gadget = new ScfSideShowGadget(s_gadgetId);
In order to add Glance content to the gadget, you
can use the ScfSideShowGadget.AddGlanceContent method. This is called from each of the
tab sheets in the sample application to update
Figure 3: SideShow Content
Creator sample application.
You create content pages with both the ScfSideShowGadget.AddContent method and the Scf.
Content method. The Scf class contains a number
of static methods that construct SCF XML strings.
In the sample application, the Content Page tab
builds a content page containing formatted text.
The code snippet below shows the AddContent
method being called to create a content page. The
method parameters are:
• An ID to represent the content ID for the
page.
• Text to place in the title bar of the page being
created.
• An array of ScfElement types that represent
the contents of the content page.
In this example, I’ve used some text and two buttons:
_gadget.AddContent(Scf.Content(
(int)contentIdNumeric.Value,
textTitleTextBox.Text,
Scf.Txt(
align, true,
txtColor, textContentTextBox.Text),
Scf.Btn(DeviceButton.Left,null,
(int)contentIdNumeric.Value-1),
Scf.Btn(DeviceButton.Right,null,
(int)contentIdNumeric.Value+1)
));
The Scf.Txt method creates a text SCF element
with alignment and color properties. The Scf.Btn
method creates buttons on the page and maps the
buttons to actions. Each button must map to a physical hardware device button and contain a target
ID. The target ID is the content ID of the page that
the gadget displays when the user pushes the button. Creating a button with a display text of null (as
in the example above) will create a hidden button
that provides a mapping between the physical device buttons and the content ID of a page to display.
The buttons in the code above provide a means to
navigate to the previous content ID and the next
content Id with the Left and Right device buttons
respectively. Figure 4 shows the sample application
and the output for a text content page.
Adding images to the content of a gadget provides
some great opportunities for simple games and picture browsing gadgets, as well as providing a richer
interface for all gadgets. Displaying an image on a
gadget page is a two-step process. You need to add
the image to the gadget contents with a content ID.
Then you can create a page that uses that image.
Adding an image to a gadget is as simple as adding
other content; you use the same AddContent method, identifying the content ID, and apply transforms
to the image and finally the image object:
88
_gadget.AddContent(
imageContentID,
ImageContentTransforms.ReduceColorDepth
| ImageContentTransforms.KeepAspectRatio
| ImageContentTransforms.StretchToFit
,imagePictureBox.Image);
Figure 4: Creating a content page.
Adding the image to a page is then a matter of using the Scf.Img method to generate the SCF for the
image to display, defining the alignment, how the
image fits on the screen, and some alternative text
(similar to how in HTML an img will use the alternative text if the image cannot be displayed). You
can view the sample application and the image page
created in Figure 5.
_gadget.AddContent(Scf.Content(
(int)ImageContentIdNumeric.Value,
imageTitleTextBox.Text,
Scf.Img(imageContentID,
align, fit, "Image cannot be displayed"),
Scf.Btn(DeviceButton.Left,
null,
(int)ImageContentIdNumeric.Value - 1),
Scf.Btn(DeviceButton.Right, null,
(int)ImageContentIdNumeric.Value + 1)
));
You can create a menu page by using the Scf.Menu
method and an array of Scf.Item elements to represent the menu items. In the sample application,
I let the user define the menu items in a ListView
control. When you create the menu page, the gadget
creates an array of ScfElement objects to represent
each menu item:
ScfElement[] menuItems =
new ScfElement[menuItemListView.Items.Count];
int menuIndex = 0;
foreach (ListViewItem item in
menuItemListView.Items)
{
int targetID =
int.Parse(item.SubItems[0].Text);
string menuText = item.SubItems[1].Text;
menuItems[menuIndex] = Scf.Item(targetID,
menuText);
menuIndex++;
}
You then use this array of ScfElement objects in the
call to the Scf.Menu method that creates the SCF
for the menu page, as shown in Figure 6:
allow the user to define the buttons in a ListView
control. The gadget creates the buttons when the
user clicks the Add Content button. As previously
stated, you must associate a button with a physical
hardware device button:
ScfElement[] dlgItems =
new ScfElement[buttonsListView.Items.Count+1];
int btnIndex = 0;
foreach (ListViewItem item in
buttonsListView.Items)
{
DeviceButton devBtn =
(DeviceButton)item.SubItems[0].Tag;
string btnText = item.SubItems[1].Text;
int targetID =
int.Parse(item.SubItems[2].Text);
dlgItems[btnIndex] =
Scf.Btn(devBtn, btnText, targetID);
btnIndex++;
}
You then use the Scf.Dialog method to
create SCF for a dialog page. In Figure 7,
I’ve shown the sample application creating a dialog page:
Figure 5: Creating an image page.
_gadget.AddContent(Scf.Menu(
(int)menuContentIdNumeric.Value,
menuTitleTextBox.Text,
ScfSelectAction.Target,
menuItems
));
The last page type that you can create is the dialog
page. A dialog page is similar to a menu page; instead of menu items the dialog page has buttons.
The sample application uses a similar technique to
89
Sidebar to SideShow
One of Microsoft’s other gadget technologies is the
Windows Sidebar. Gadgets written for Windows
Sidebar can create and show content in Windows
SideShow using a set of APIs provided in the Windows Sidebar SDK. The APIs take raw SCF input so
an author of a Windows Sidebar gadget pushing content to Windows SideShow needs to be very careful
to get the format correct. A useful tool is a Windows
Sidebar gadget that provides a way to test the SCF
being sent to the Windows SideShow gadget. You
can see this in action in Figure 8 and download it
from http://www.roodyn.com/sidebarsideshow.ashx
Figure 6: Creating a menu page.
The Windows Sidebar API is possibly the easiest
way to create content to display on a Windows
SideShow device. The downside is the resulting
Windows SideShow gadget is tightly coupled to the
Windows Sidebar gadget. If the Windows Sidebar
gadget is closed, the Windows SideShow gadget is
no longer shown or available.
_gadget.AddContent(Scf.Dialog(
(int)dialogContentNumeric.Value,
dialogTitleTextBox.Text,
dlgItems));
The Managed API makes creating gadgets and adding content
a simple exercise. The Managed API contains classes and
The Managed API makes
methods that enable a gadget
creating gadgets and adding
to do far more than simply add
content. There are classes and
content a simple exercise.
methods to query a device for
its capabilities, show notifications on the device, raise events
when content is missing, and far
more than this article can cover. You can find the
full API documentation on MSDN at http://msdn2.
microsoft.com/en-us/library/ms744179.aspx
The Windows Sidebar API for Windows SideShow consists of the System.Gadget.SideShow
namespace, which contains five methods, one property, and one event. As you do the implementation
of a Windows Sidebar gadget in JavaScript, it is important to remember the limitations of this environment. While powerful, it is not strongly typed and it
is easy to make costly mistakes.
To determine if a Windows SideShow device is
available before adding content, you should use the
enabled property. Although called a property, it is
implemented as a method call, so remember the
brackets after the method call:
if(!System.Gadget.SideShow.enabled())
{
contentText.value = "SideShow device not found";
}
Once you have determined the presence of a Windows SideShow device, you create a Windows SideShow gadget by calling the setFriendlyName method. This method takes one parameter that is used
as the display text for the gadget in the Windows
SideShow Control Panel as well as on the device:
Figure 7: Creating a dialog page.
System.Gadget.SideShow.setFriendlyName(
"Dr. Neil's SideShow Test");
To add Glance text to the Windows SideShow gadget, you should use the addText method with the
content ID set to 0. Glance text is just a text string
and neither SCF nor any other markup is required
(nor should be used):
System.Gadget.SideShow.addText(0,
"Test your SCF here");
You add an image with the addImage method. This
takes as parameters the content ID and a file path
to the image. This method supports JPG, GIF, and
BMP formats for images. The images will be scaled
90
Figure 8: Sidebar gadget creating content in a SideShow gadget.
The Windows Sidebar API
is very simple and does not have
the full feature set of the
Managed API or COM API,
but it is a quick way to get
content out onto the
Windows SideShow device.
to display on the device; this might lead to some
degradation of the image quality:
var path = System.Gadget.path;
var imgFile = path + "\\code.gif";
System.Gadget.SideShow.addImage(100, imgFile);
Using the addText method, you can add raw SCF
to the gadget content. Unlike most modern browsers, the Windows SideShow rendering engine is
very unforgiving—the slightest error in the SCF will
prevent the entire page from loading. This is one
reason I wrote this gadget to test the SCF for Sidebar gadgets:
has huge potential. It allows applications that are
running on a PC to place content on other devices
that can viewed even when the PC is switched off.
When choosing which development tools to use
for building a Windows SideShow gadget, consider
the depth of experience you would like to provide.
The COM API is the most powerful toolkit and,
like many power tools, it can also be dangerous.
The developer needs to have a strong COM skill
set and understand the implication of threading and
memory management. The Managed API is almost
as powerful as the COM API and relieves the developer from much of the drudgery regarding formatting of content such as SCF. The Windows Sidebar
API is a great tool for simple Windows SideShow
gadgets that extend the information from a Windows Sidebar gadget.
Whichever API you chose, placing content onto a
Windows SideShow gadget will prove a rewarding
experience that enables the users to access application information via another user interface.
…placing
content onto
a Windows
SideShow gadget
will prove a
rewarding
experience that
enables the
users to access
application
information
via another user
interface.
Neil Roodyn
var txt =
"<body><content id=\"3\" title=\"The Title\">";
txt += "<txt align=\"c\" wrap=\"0\">Test</txt>"
txt += "</content></body>"
System.Gadget.SideShow.addText(1, txt);
The Windows Sidebar API is very simple and does
not have the full feature set of the Managed API or
COM API but it is a quick way to get content out
onto the Windows SideShow device.
Final Thoughts
Developing Windows SideShow gadgets can be a
rewarding experience. It is a new technology which
91
Tablet PC Input Panel
Programmability
Jan-Kristian
Markiewicz
[email protected]
Jan-Kristian recently moved
from Norway to work on the
Tablet PC team in Redmond. As
a program manager on the Input
Panel team, he is responsible
for driving new improvements to
pen text input on the Tablet PC
platform.
While attending the Norwegian
University of Science and
Technology and the University
of California, Santa Barbara,
he received acknowledgement
as both a Microsoft Student
Ambassador and as a Microsoft
Student Partner. He has
participated in the Imagine Cup
student software design contest
2004-2006, which has taken
him to the world finals in both
Brazil and India. In India his team
placed third with their solution for
monitoring and distributing health
parameters.
92
Tablet PC Input Panel Programmability
Most applications will require some form of text input. By using
the ITextInputPanel API you can optimize the pen text-input
experience on Tablet PCs.
If you have ever used a Tablet PC, you have most likely used Tablet PC Input Panel.
Input Panel is the handwriting input tool that lies on top of your application so
that you can insert handwritten text into any non-ink enabled Microsoft® Windows®
control. Using Input Panel programmability is a great way to ensure that Input Panel
provides the best possible pen input user experience in your application.
U
sers can choose from two main ways to acthing about setting preferred behavior is that Input
cess the Input Panel in the Windows Vista™
Panel will still make sure that it is always in a usoperating system. You can either tap the Inable state. For example, if you set Input Panel to
put Panel tab, which is located on the left or right
preferably appear above a certain text box and the
edge of screen, or you can tap the icon that appears
user has moved the window to where the text box is
whenever you use your pen to
located at the top of the screen,
give focus to a text field. When
Input Panel will automatically
Fast Facts
you launch Input Panel using
make sure that it is visible by
Input Panel allows users to
the second approach, Tablet PC
placing itself below the text box
insert handwritten
uses several parameters to deinstead. In addition to setting
cide how to display Input Panpreferred behaviors, you can
text into any non-ink enabled
el, such as the placement of the
manually specify Input Panel’s
Windows control.
text box relative to the screen
exact screen position. When
By using the ITextInputPanel
edges, the size and text entry
you do this, Input Panel places
API you can easily modify the
mode from the last time you
itself exactly where you specify;
behavior of Input Panel so
used Input Panel, and whether
therefore, you are responsible
that it works optimally with
the text box is a password field.
for positioning it on the screen
your application.
Most of the time Input Panel
and making sure it is visible.
will do a good enough job with
this, but what if you want Input
Panel to interact in a more customized way with
Programmability Possibilities
your application? This is where Input Panel programmability comes into the picture.
Using the ITextInputPanel API is a great way to
make your application work even better on Tablet
In Windows Vista you can use the new ITextInPCs. I’ll quickly summarize some of the main things
putPanel API, with either unmanaged or managed
the API allows you to do before I take a look at
code, to tell Input Panel how you want it to behave
how to use it:
around the controls in your application. The properties with “Preferred” in their name allow you to
• Set absolute position of Input Panel.
set preferred behavior for Input Panel. The nice
• Set preferred direction—you can set the preferred direction of where Input Panel should
appear relative to your controls (on the top or
bottom of the control).
• Set visibility—you can choose not to show InFigure 1: When the user gives focus to the “Age”
put Panel for certain controls.
text box, the Input Panel will automatically expand
• Set expanded state—make Input Panel go
and change to keyboard mode.
straight to the expanded state when a control
gets focus.
• Set default input area—select any of the three
input areas as the default for a control (writing pad, character pad, or on-screen keyboard).
• Look up Input Panel properties such as size,
current input area, or current correction
mode.
• Listen to event changes of the Input Panel
state.
• Collect Ink objects through Input Panel in addition to the recognized text.
So what can you do with all of this? You can make
sure that Input Panel doesn’t cover essential parts
of your user interface while it is in use, for example.
Or you can set Input Panel to expand as soon as
your controls get focus, which saves pen input users time and effort each time they want to add text
to a field. If you have a text field where the users
always enter a mix of characters and numbers that
you know isn’t a good fit for handwriting, e.g., a
license plate number, you can automatically switch
Input Panel to keyboard mode and save the users
the hassle of switching back and forth.
The ability to capture ink and recognition results as
they are sent from Input Panel to the controls in your
application also opens up a lot of possibilities. You
can analyze or save users’ ink data from Input Panel
or you could even alter their input, for example, to
automatically expand shorthand into full words.
Using the ITextInputPanel API
Figure 1 shows a sample where I’ve added some
custom Input Panel behavior to a simple form. I set
all three text boxes to interact differently with Input Panel. For the first text box, I want Input Panel
to appear over the box, so that Input Panel doesn’t
cover the text boxes below. I also want it to expand
as soon as the control gets focus. I will set the second text box to always hide the Input Panel when
it gets focus. This could be useful if users enter the
ID# using another input method, such as a barcode
scanner. Finally, I will set the last text box to expand
Input Panel in keyboard input mode so that the user
can quickly enter the age using the number keys.
The code in Listing 1 shows how you can achieve
the wanted behavior by using C# and the managed
ITextInputPanel API. The first thing you have to
do to be able to use Input Panel programmability
features is to add a reference to the Microsoft.Ink
DLL. It should appear in the list of .NET components under the name “Microsoft Tablet PC API”
(version 6.0 or higher). Next add a using statement
for Microsoft.Ink.TextInputPanel, and then you’ll
be ready to use the TextInputPanel object to manipulate Input Panel.
Corinne Sherry
User Experience Designer,
Mobile & Tailored Platforms
Division
Conclusion
I hope this article has given you a little insight into
the possibilities of the ITextInputPanel API and how
you can utilize it to make sure your Tablet PC application provides a great Input Panel user experience.
Customizing your application to work well with Input
Panel has many advantages, especially if you know
that many of your clients will use Input Panel to enter
text. With only a few lines of code you can make big
user experience improvements. Maybe your application works great 90% of the time with the Input Panel
default settings, but making sure it works perfectly for
the last 10% can make the difference between a user
and a happy user.
I enjoyed working on the
Windows Vista Mobile PC User
Experience Guidelines. It’s
exciting to think of the impact
we can have on mobile PC
users’ experience by helping
software developers optimize
their programs for mobile use.
I’m sure the next generation of
mobile programs will be better
than ever.
Listing 1: C# code for a simple form with custom Input Panel behavior
using System;
using System.Windows.Forms;
using Microsoft.Ink.TextInput;
InPlaceState.Expanded;
// make Input Panel disappear each time the
// ID text box gets focus
tipID = new TextInputPanel(textBoxID);
// the event handler will hide Input Panel for you
textBoxID.GotFocus +=
new EventHandler(textBoxID_GotFocus);
namespace sampleTIP
{
public partial class Form1 : Form
{
// declare the TextInputPanel objects
TextInputPanel tipName = null;
TextInputPanel tipID = null;
TextInputPanel tipAge = null;
// set the default input area to keyboard and the
// state to expanded for the ID text box
tipAge = new TextInputPanel(textBoxAge);
tipAge.DefaultInputArea = PanelInputArea.Keyboard;
tipAge.DefaultInPlaceState = InPlaceState.Expanded;
}
catch (Exception ex)
{
// error handling
}
}
public Form1()
{
// initialize the TextInputPanel objects inside
// a try/catch block since the objects aren't
// supported in all versions of Windows
Try
{
// set the preferred direction to top and the
// state to expanded for the name text box
tipName = new TextInputPanel(textBoxName);
tipName.PreferredInPlaceDirection =
InPlaceDirection.Top;
tipName.DefaultInPlaceState =
// hide Input Panel each time the ID text box gets focus
void textBoxID_GotFocus(object sender, EventArgs e)
{
tipID.SetInPlaceVisibility(false);
}
}
}
Tablet PC Input Panel Programmability
93
Exploring Tablet PC
Application Deployment
Rod Paddock
Rod Paddock is the Editor-inChief of CoDe Magazine.
Rod is a Microsoft VB.NET MVP
and develops software with tools
like SQL Server, Visual Basic, C#,
ASP.NET, and numerous others.
Rod is president of Dash Point
Software, Inc, an award-winning
software development firm
that specializes in developing
applications for small to large
businesses. Dash Point has
delivered applications for
numerous corporations including:
Six Flags, First Premier Bank,
Intel, Microsoft, and the US Coast
Guard.
You can find Rod’s blog at
http:\\blog.dashpoint.com
You have decided to take the plunge and create a Microsoft®
Windows® XP Tablet PC Edition operating system-aware
application. This decision comes with a new set of requirements when it comes to
enabling Tablet PC-specific features and deployment of your application. This article
will take you through the process of creating a Tablet PC-aware application and
deploying it in the enterprise.
W
hen developing Tablet PC-enabled applicaThe real limitations of your operating systems prestions you must take a number things into
ent themselves when it comes to deployment. For
consideration. Do you create applications
instance, Windows XP Professional or Windows
specific to just the Tablet PC
2000 don’t natively support
platform? Do you create a single
ink recognition. These operatFast Facts
application that houses “staning systems can collect ink but
Developing and deploying
dard” application inputs and
there are no default recognizers
Tablet PC-enabled
then augments that functionality
installed to turn that ink into
with tablet features where reletext. To turn ink into text you
applications have a number
vant? How do you know if you
will need Windows XP Tablet
of unique aspects.
are running in a Tablet PC-enPC Edition 2005, a Premium
This article delves into both
abled environment? What tools
edition of Windows Vista, or a
of these topics.
do you need to install to develcopy of Windows XP with Miop these applications? What do
crosoft Windows XP Tablet PC
you need to do to deploy these applications into the
Edition 2005 Recognizer Pack installed (see sideenterprise? This article will address these issues.
bar, Recognizer Pack).
The Tools
Visual Studio 2005 developers who want to develop pen-enabled applications don’t have to do
a lot more than they’d do to build any application
with the .NET Framework. You can develop penenabled applications using the following operating
systems and toolsets:
•
•
•
•
•
•
•
Windows Vista™ Premium Editions
Microsoft® Windows® XP Professional (SP2)
Microsoft Windows Server™ 2003
Windows XP Tablet PC Edition 2005
Microsoft Windows 2000 (SP4)
Microsoft Visual Studio® 6.0 (SP5)
Visual Studio .NET (2003 or 2005)
After installing/acquiring the proper operating systems and tools you need to download and install
the Tablet PC SDK from the Microsoft Web site.
You can develop and test your applications using
any of the above operating systems provided you
have the proper tools installed. (For a more detailed
explanation of development environments for Tablet PC-enabled applications, see Eliot Graff’s article,
“The Proper Developer Environments for Mobile
PC, Tablet PC, and Ultra-Mobile PC Applications,”
found in this issue of CoDe Focus.)
94
Exploring Tablet PC Application Deployment
After installing the Tablet PC SDK you can get to
work on pen-enabling your application. This article will use Windows XP Professional and Visual
Studio 2005 to create a simple Tablet PC-aware
application and then deploy that application to
Windows Vista Ultimate and Tablet PC Edition
2005.
Tiny Tablet Application
My small Tablet PC-enabled application has the following requirements:
• Capture ink input from the end user.
• Determine whether the platform supports Tablet PC functionality.
• Turn captured ink into text when your operating environment is capable.
• Determine what recognizers are available to
an application.
To build this application, do the following:
1. Create a new Windows application.
2. Add a reference to the Microsoft.Ink and Microsoft.Ink.Analysis assembles. You can find
these assemblies in the following location:
“C:\Program Files\Microsoft Tablet PC Platform SDK\Include\Microsoft.Ink.dll”
3. Add three buttons to your form. Name them
cmdRecognize, cmdClear, and cmdShowRecognizers respectively. Set the Text property of
these buttons to Recognize, Clear, and Show
Recognizers, respectively.
4. Add a Panel control to your form. Name it
pnInk.
5. Add an imports statement referencing Microsoft.Ink assembly to your form.
6. Add an overlay member variable to your form.
7. Add the following code to the Load event of
your form.
MyCollector = New InkCollector(Me.pnInkHandle)
8. Add the following code to the Click event of
the cmdRecognizeButton.
Dim cString As String = _
Me.MyCollector.Ink.Strokes.ToString
MessageBox.Show(cString)
9. Add the following code to the cmdClear button’s Click event.
Me.MyCollector.Ink.DeleteStrokes()
Me.pnlnk.Invalidate()
Now run your application. Your Panel control
should accept ink input. However, when you click
the cmdRecognize button you may receive an exception if you’re not on a Tablet PC computer because there are no recognizers installed. You will
need a Tablet PC-enabled OS to support ink recognition. The next section will demonstrate how to
determine whether Tablet PC features are available
to your applications.
Pen Enabled
The first step to building a Tablet PC-aware application is to determine whether or not you are running
in a Tablet PC operating environment. You can test
whether you are running in a Tablet PC-enabled environment by calling the Windows API GetSystemMetrics function with a value of 86. For example:
Module WindowsAPI
Private Declare Auto Function
GetSystemMetrics Lib "user32" _
(ByVal value As Integer) As Integer
Private Const SM_TABLETPC As Integer = 86
Public Function IsTablet() As Boolean
Dim lnTestValue As Integer = _
GetSystemMetrics(SM_TABLETPC)
Return lnTestValue > 0
End Function
End Module
When executed from a Tablet PC 2005 environment this function always returns true. Windows
Vista Premium environments
have a different set of requirements when determining Tablet PC capabilities. Windows
Vista Premium editions will
only return true if a number of
preconditions are met:
• A digitizer or pen device is
installed on the machine.
• The version of Windows
Vista must be licensed for
Tablet and Touch features
(i.e. a premium version of
Vista).
You may receive
an exception if you are on a
non-Tablet PC-enabled
computer because there are no
recognizers installed.
You will need a Table PC-enabled
OS to support ink recognition.
During my development I found something strange.
All of the features of my application worked on
Windows Vista Ultimate even though the GetSystemMetric call I made returned false. I had forgotten to install my digitizer on my desktop machine. I
decided to use this inconsistency to my advantage.
I determined that another mechanism for determining whether or not your computer has Tablet PC
features installed is to see if there are any Recognizers installed. Recognizers are the APIs that Tablet
PC-enabled operating systems use to turn ink into
text. Tablet PC-enabled operating systems include a
number of these recognizers.
On my Windows Vista machine, I knew that the
Tablet PC features were installed but the GetSystemMetrics function always returned false. I used
the Recognizers collection to determine whether
my machine was actually tablet-aware.
In the previous section I pointed out that your application would blow up if you ran the code on a
machine lacking Tablet PC features (i.e. recognizers). I changed my recognize code to look for any
recognizers installed on the machine.
Dim recognizers As Recognizers = New Recognizers()
If recognizers.Count > 0 Then
Dim cString As String = _
Me.MyCollector.Ink.Strokes.ToString
MessageBox.Show(cString)
Else
MessageBox.Show("No recognizers installed")
End If
Note: Standard editions of Windows Vista have a
single recognizer installed on them (gesture recognition) so you may want to change the count you
test for to > 1.
Recognizers are the
APIs that Tablet PC-enabled
operating systems
use to turn ink into text.
The Tablet PC
SDK 1.7
You can find the
Tablet PC SDK at:
http://www.microsoft.
com/downloads/details.
aspx?FamilyID=69640b5c0ee9-421e-8d5c-d40debee36c
2&displaylang=en
Recognizer Pack
If you want to convert
handwriting from digital ink
to text, in multiple languages,
download the Windows XP Tablet
PC Edition 2005 Recognizer Pack:
en-us/library/aa813362.aspx
Determining if
a Machine Is
a Tablet PC
If you want your application to
determine whether or not it is
running on a Tablet PC, look at
this site:
ClickOnce For
the Real World
Not Hello World
Julia Lerman goes in depth
into the world of ClickOnce
deployments.
http://code-magazine.com/
Article.aspx?quickid=0611041
95
ing applications using the two most common types
of deployments: Setup.EXE and ClickOnce deployment.
Creating a Setup Project
I’ll first describe using a Windows setup deployment
scenario. Follow these steps to create a Tablet PCenabled application setup program:
Figure 1: Creating a new Setup Project in Visual Studio 2005.
You can now complete the final requirement of
the application requirements listed earlier. The
application will determine what recognizers are
installed in the operating system. The following
code lists the recognizers installed on your Tablet
PC.
Dim recognizers As Recognizers = New Recognizers()
If recognizers.Count > 0 Then
For Each oReco As Recognizer In recognizers
MessageBox.Show(oReco.ToString)
Next
Else
MessageBox.Show("No recognizers installed")
End If
The last two examples illustrate enabling Tablet PC
features even when the operating system returns a
value that might not be true.
If you want to make sure that you can take advantage all Table PC features you could use a strict
approach and only
enable
features
when the IsTablet()
you created earlier,
returns true. Otherwise you can use a
less stringent check
like the one in the
prior snippets.
Deploying Your
Applications
Figure 2: Specifying .NET Framework as a prerequisite for your setup.
96
Once you have created your applications you need to
ship them out to users. This section will
demonstrate deploy-
1. Right-click on your solution in the Solution
Explorer and select Add from the shortcut
menu and then choose New Project. The New
Project dialog box will open.
2. Select Other Project Types and then choose
Setup and Deployment.
3. From the templates section select Setup Project. Name your project TabletPCEnabledSetup. See Figure 1.
4. Right-click on your new setup project in the Solution Explorer and select Add from the shortcut menu, and then choose Project Output.
5. Select your tablet application project from the
dropdown list of applications. By default it will
specify the primary output of your application
as the item to install.
6. At this point the setup application will interrogate your application and determine which
DLLS and other files it will need to install.
When creating your installer you should use
the installation .msm files provided in the Tablet PC SDK. The basic .msm files you will need
are MSTPCRT.DLL and TPCMAN17.MSM.
For additional functionality, consult the Tablet
PC SDK.
7. You may need to specify whether or not to
install the .NET Framework if it is not present on the target machine. To do this you need
to specify the prerequisites for your setup application. You specify prerequisites from the
Property Pages of your setup applications.
8. Right-click on your Setup Project. Select Properties. From the Property Pages select the Prerequisites button.
9. The Prerequisites dialog box selects the .NET
Framework 2.0 by default. If not select it in the
list of options.
10.If you want the .NET Framework to travel
with your installer you need to make one more
change. Specify the “Download prerequisites
from the same location as my machine” option. See Figure 2.
11.Build your application by right-clicking on the
setup project and select Rebuild.
To test your application, copy the contents of your
setup project’s output directory to the target machine and run the Setup.EXE file created by the
build process. If Setup.exe doesn’t detect .NET
Framework 2.0 on the target machine, the Setup
program will offer to install it. After installing the
.NET Framework, the setup program will install
your application code. You can now test the application from the appropriate Windows Programs
menu.
Building a setup application is a pretty simple task
but not very efficient when it comes to distributing
that application across the enterprise. For enterprise
deployment, consider using ClickOnce features in
Visual Studio 2005.
Creating a ClickOnce Deployment
One of the cool features of Visual Studio 2005 is its
ability to deploy applications via the Internet using
a technology known as ClickOnce. ClickOnce deployment greatly simplifies shipping applications to
a large number of workstations. In the next section
I will demonstrate how to deploy your applications
using ClickOnce.
You specify the options for ClickOnce deployment via the properties sheet of your application.
To specify ClickOnce deployment options, open
the property pages of your application and select
the Publish tab (Figure 3). Now click Publish
Wizard…. The wizard will walk you through a
number of steps where you’ll specify which Web
site you want to publish your application to and
what Web address you want users to install your
applications from. Specify the appropriate settings for your particular environment. The simplest mechanism for testing a ClickOnce deployment is to have it publish its deployment code to
your local IIS installation and test it from there.
After testing your ClickOnce deployment from
your local machine you can deploy it to an internal Web server or public Web site by specifying
the appropriate settings for Publish Location and
Installation URL.
You have to make one required change in IIS for
the public Web folder you deploy ClickOnce items
from. You need to make sure that IIS enables ASP.
NET functionality in that folder. The following
steps demonstrate enabling ASP.NET in a Web
folder.
1. Open IIS. Navigate to your ClickOnce folder.
2. Right-click and select Properties… from the
shortcut menu.
3. If the Application Name field is disabled, click
the Create button and select Apply (Figure 4).
If you do not specify these properties the ClickOnce
installation will fail when the user attempts to install you application from that ClickOnce deployment folder.
In the setup project created earlier you specified
that your setup project should install the .NET
Framework if it is not already present on the target
machine. This option is also available in a ClickOnce deployment. You specify this option by clicking on the Prerequisites button and selecting the:
“Download prerequisites from the same location as
my machine” option.
After specifying the prerequisites for your ClickOnce deployment you are almost done. You need
to specify one more set of options.
Eliot Graff
Manager, Developer
Experience, Mobile & Tailored
Platforms Division
I can’t believe that I get to help
developers embrace a whole
new arena of application use.
Full computing—anytime,
anywhere—is the new order of
business.
By default the ClickOnce deployment you create
will most likely fail to run on a non-Tablet PC ma-
Figure 3 demonstrates publishing a ClickOnce
application to a UNC directory. This directory is
actually a Web site subfolder dedicated to ClickOnce deployments. The Installation URL properties direct the user to install from a public Web
site.
Figure 4: Enabling ASP.NET on a Web folder.
Figure 3: ClickOnce publishing Properties Page.
97
.N
ET
Advertisers Index
Developing Windows Vista™ applications for notebooks, Tablet PCs, and UMPCs
2007 Volume 4 / Issue 2
Build Ultra-Mobile
Applications
Windows Vista
Mobility
Develop for
Windows
SideShow
WPF on
Mobile PCs
w w w. c o d e - m a g a z i n e . c o m
US $ 5.95 Can $ 8.95
Advertising Sales:
Vice President,
Sales and Marketing
Tom Buckley
832-717-4445 ext. 34
[email protected]
Sales Managers
Erna Egger
+43 (664) 151 0861
[email protected]
Figure 5: ClickOnce deployment Web page for your Tablet PC application.
chine. If you are creating an
application that is supposed
to run on Tablet PC and nonTablet PC machines you will
need to install the ink assemblies and supporting files on
the target machine. Unfortunately there is no ClickOnce
mechanism available to install
these libraries on a non-Tablet
PC. When deploying these applications to a non-Tablet PC
you will need to install the tablet features on the target machine using a separate setup
program. The setup program
you read about earlier in this
article will do the trick.
Now you can re-publish your
ClickOnce deployment and it
should install all of the necesFigure 6: Tablet PC-enabled application running in Windows
sary files to run your applicaVista Ultimate.
tion. Figure 5 shows the installation page for this
sample application and Figure 6 shows the actual
running application in Windows Vista Ultimate.
Rod Paddock
Tammy Ferguson
832-717-4445 ext 26
[email protected]
Advertisers Index
Active Ink Software
www.activeinksoftware.com
63
CoDe Magazine
5
EPS - Electronic Publishing Platform
www.code-magazine.com/ripple
EPS Software Corp.
www.eps-software.com
100
7, 36-37
Microsoft Mobile & Tailored
Platforms Division
msdn.microsoft.com/mobilepc
2-3, 11
StrataFrame
www.strataframe.net
61
Tech Conferences Inc.
www.devconnections.com
50-51
VFP Conversion
www.vfpconversion.com/tools
70-71
West Wind Technologies
www.west-wind.com
99
This listing is provided as a courtesy to our readers and
advertisers.
The publisher assumes no responsibility for errors
or omissions.
98
Our Next Generation
Digital Publishing
Platform will cause
quite the ripple!
www.code-magazine.com/ripple

Mobile PC Development

Transcription

Similar documents

your tax return and save a tree.

HM-6106 - Gravograph

The New $100 Bill is Here

Charfoos Design

PDF Flyer - PyroGraphics

INDUSTRIAL INKJET PRINTER

10290 DTG Spec sheets2

- Color

Benefits of Pigment Ink for Wide-format Printing

Maximum Media / Print Width