How to get started on your own FTIR Multi-touch device Chris Delbuck

How to get started on your
own FTIR Multi-touch device
Chris Delbuck
Spring 2007
DAI627: Digital Media 4
Prof. Paul Catanese
Some things you should know first.
Multi-touch is a term used for a touch based interface system that allows for multiple
input points (in this case derived from touching a surface). Some of the inclusive
benefits of a multi-touch system are: 1) suitable for multiple users 2) freedom from the
cursor 3) higher quality input.
FTIR: Frustrated Total Internal Reflection (or the frustration of Total Internal Reflection)
This is the primary physical property that allows for this multi-touch approach to be
possible. Total Internal Reflection is an “optical phenomenon” that occurs in
transparent materials, in this case clear acrylic. Different mediums reflect, bend or pass
light depending the angle at which the light approaches the surface. In this usage we
want to keep the light utilzed for this method trapped inside the acrylic. The frustration
occurs when a finger or other material with a higher refractive index come in contact
with the medium conducting the total internal reflection.
Acrylic: For the FTIR Multi-touch approach, clear acrylic sheet (”plexiglass”) is an
optimal medium to use to achieve total internal reflection. However there are some
individuals who have managed to achieve the TIR with polycarbonate materials. 1/4”
thick Acrylic sheet is preferable because it will remain rigid enough for touch
interaction and will be narrow enough to create more dense internally reflected light.
Infrared Light (IR): Since this approach will use computer sourced projection and
computer vision it is important to use a combination visible and non-visible (or near
visible) light with each being designated to one of the components. The visible light
will be reserved for display so the user has visual feedback to their engagement with
the device. The IR (non-visible) will be used for the total internal reflection and a video
camera to capture the points (touches) where the light is frustrated. IR light is light that
exceeds a 750nm wavelength (visible light exists from 380 - 750nm). Since we cannot
see it, the IR light can exist simultaneous to the visible light of the projection. The
range of IR light used for this can range from 750 - 950nm, the decision will be made of
the application and setting of its use and surrounding ambient light.
A list of essential materials.
Acrylic Sheet: 1/4” in thickness is optimal. The greater the thickness, the less number of reflections
that occur, though this can be compensated with more IR light (would not suggest anything over
1/2“). Anything below 1/4” becomes potentially too flexible for touch and counterproductive to
successful total internal reflection. Width and height are completely up to the application and usage
of the multi-touch system.
Camera: A CCD based camera that has the ability to send a digital signal to the computer is
necessary to capture the points of frustrated light. A black and white camera is absolutely fine if not
optimal for this application because more precise capture from black to white (or 0.0 - 1.0) can be
achieved. Some USB webcams can be used for this, however higher quality Firewire based cameras
are optimal.
Lens: A lens that does not have an IR coating or filter integrated into it. Some webcams have a small
layer of film that acts as an IR filter which can be removed.
Visible Light Blocker: A means to prevent visible light from entering the camera is necessary, only
the frustrated IR light from the acrylic is required to be captured. Common 35mm photographic
negative film is a perfect material. Pull out any length from the roll (cartridge) and expose to direct
sunlight for at least 5 - 10 minutes, then have the film processed at any lab. The returned material
should be completely visibly opaque and dark in nature. Cut the film negative to size to fit between
the lens and the image sensor of the camera (sometimes double layering is necessary).
Projector: For the user to have a direct visual correlation to where their fingers engage the surface,
a projection onto that surface is necessary. Any typical computer sourced projector should work in
this application. Keep in mind different projectors have varying angles, so calculate the desired size
of projection to the require distance the projector must have from the surface. If you are projecting a
4:3 ratio image you will probably want a piece of acrylic that will have a visible area also close to 4:3.
IR LEDs: Light Emitting Diode’s will be the light source for the total internal reflection within the
acrylic. You can choose LEDs with a visible light range from 750-950nm, depending on how much IR
light might be ambient in the area. 840-850nm is typically a safe option. LEDs are typically 5mm in
diameter, only go larger if the acrylic is thicker than 1/4”. The most important specification you must
look for is the beam angle. Total angle of 80° is the maximum allowable for this process. Make sure
when purchasing of the beam angle is referenced as total or half angle, half angle should be 40°.
You will also need an adequate power source, resistors and length of quality copper wire for the LED
configuration you choose. The number of LEDs needed depends on how large the area of touch
surface is required. To achieve a high level of light, use one LED for every inch on the vertical (left
and right) edges.
Computer: One that can interface with the camera and projector being used. As well as adequate
enough to process the video signal and drive a resulting projection. Two computers can be used,
one for each task, with some communication bridge between them.
Materials subject to variation.
Framing and Structure: Something will be required to hold the Acrylic in place as well
as hold the IR LEDs flush to the edges of the acrylic. Either wood or metal can work.
Something sturdy enough to hold up the acrylic and keep it in place when used. I
found some very cheap wood frames at IKEA that worked perfectly. I lined up the
acrylic in between the two frames, secured the pieces with tape and clamps, then
drilled through all three pieces uniformly. This allowed me to secure all pieces with four
bolts and nuts.
Adhesives: Some various tapes and glue are always helpful to keep things in place
either permanently or temporarily.
Projection Surface: One of the largest areas subject to variation is how to place the
projection onto the relatively transparent surface. Since the nature of TIR is subject to
frustration, its important to have a surface for projection that does not set off the FTIR
effect. If between the projector and the acrylic, allows for IR light to pass through.
Other Surfaces: Since bare acrylic is not necessarily the easiest to drag fingers across it
becomes troublesome as an effortless touch surface, in fact the FTIR method requires
some force from the fingers to activate. Ideally a “zero-force” approach with easy
gliding would be the best for a worthwhile experience with the multi-touch system.
FTIR in further depth.
As light passes into the sides of the acrylic, it will remain reflected as long as the angle
which it approaches the surfaces is less than 42.1° (the angle of total internal reflection
of acrylic to air). Once something comes in contact to the surface of the acrylic with a
higher refractive index, the internal reflection is frustrated and light will leave the acrylic
at that point.
Acrylic
LED
Internal Reflection
Light beyond
critical angle
Frustrated Light
Projection and other material layering.
In my work I ended up using 4mm thick sheet of Mylar with a single matte side. This
could lay flat against the back side of the acrylic and not set off the FTIR, it is also a
very nice projection material as well.
Some individuals have found a combination of layers that achieve this. Silicone Rubber
and Rear Projection Screen allow for easy gliding and near zero-force FTIR. Utilizing
clear silicone rubber (either a poured from a two part curing compound or pre-made
sheet) no more than 1mm thick directly on the topside of the surface and then gray rear
projection screen material on top of that seems to be one of the best configurations for
an easy to use multi-touch system. The nature of the Silicone Rubber allows for a great
deal of projected light to pass through, and only sets of the FTIR upon touch. The
Projection Screen delivers great image quality from the projection and as very easy
surface to engage with fingers.
My Approach
Another Approach
Projection Screen
Silicone Rubber
Acrylic
Mylar
Acrylic
Projection
Projection
LED configuration and
acrylic edge preparation.
Various approaches can be tackled in order to conduct TIR of IR light within the acrylic
medium. Some individuals arrange LEDs on all 4 sides of the material, while others
only do 1. I decided to do the left and right sides of the surface, placing an LED at
every inch in the length of the surface that would be utilized for a touch area.
What is also very important, is the preparation of the acrylic edges. Typically when the
acrylic is cut with a saw, the edges are left very rough, which is not optimal for light
transmission. There are various approaches to purifying the edge of any grain or
roughness and have an appropriate clarity for the IR LEDs. One option is to use various
grades of sand paper (in the order by grit, 100, 200, 400 and then 800) then polishing
with either copper or brass polish. With mine, I opted to have the supplier use a router
and flame polish the edges. This produced a very good edge quality for the infrared
light.
Hardware configuration.
Most calibration to achieve an accurate visual correspondence to the physically
touched area will occur in software, it is important to understand the relation where all
of the components will exist. Though this is not the only configuration possible (mirror
to alter placement can work) this is the arrangement in basic principle. The camera and
projector must be placed on the opposite side of the surface from the user, being able
to project and capture, visible and IR light respectively.
12VDC
IR LIGHT
IR CAMERA
VIEWPOINT
FRUSTRATIONS
DIGITAL
PROJECTOR
SENDS IMAGE
TO PROJECTOR
IMAGE ANALYSIS
SENDS IMAGE
TO COMPUTER
VISUAL
GENERATED
COMPUTER
ESCAPED LIGHT
Image Analysis & Visual Generation.
For this proof of concept, it is necessary to illustrate the connection between where a
user touches and how the device visually reacts to engagement. It is necessary for the
furstrated areas (touches) to become seen by the IR sensitive camera. Then that signal
is sentto the computer. The image is analyzed and based on development can include
calibration, the resulted analysis is the generation of a visual response back to the
surface where the user interacts with it.
The left is a still from the IR Camera
The Right is the projected image
Image Analysis & Visual Generation.
(cont.)
The image being delivered from the camera to the computer is a
grayscale image that will have white spots on a mostly dark
background. The white spots come from where the internal reflection
of the IR light in the acrylic is frustrated. The camera captures the
touch areas as bright white blobs. These blobs captured by the
camera are the main component to how the computer will interpret
the image and generate a resulting visual response to the touches.
Using blobDetection by v3ga (processing library), I was able to capture
the image coming from the camera. In addition to camera capture, the
software also allows for finding bright areas apart from dark areas in
the image. These blobs are found based on areas in the image that
fall above a set brightness level. Each blob detected is analyzed and
has certain attributes computed. Each blob has an overall width and
height, which is used to also find a center. The edge of the blob is
broken up into edge points. The software can return the edge points
one at a time, or two at a time to be used to create a line. There is
also the functionality to access blobTriangles which takes two edge
points at a time and the center point to return three points which can
be used as vertices.
Amongst the aforementioned types of data the blobDetection will
return there are others as well. In most cases, centers will typically be
the primary information be used, just to most accurately pin-point
where the user is interacting. However the edges and triangle
information could easily be used as a means to interpret pressure
being applied by the user. Perhaps a blob generated by the palm of a
hand touching the surface could create a different response than a
fingertip.
The Final Product.
(Conclusion)
Essentially the goal of this project was to at least get to a stage where the hardware was working
with the FTIR approach and generating something basically responsive. This was achieved, and
even on a simple level of reaction, still was very engaging for the small audience that was the first
to witness the project in its “completed” form.
Going through two prototypes in hardware was definitely a consumer of energy and work hours.
The second prototype (shown below) was much more managable since I used prefabricated
framing materials which were modified for this project. If this had been done as part of a group
project with the knowledge and resources of the product designers in DAI, the end product could
be something more substantial and applicable for public use.
The possible software development for this hardware approach is almost endless. Since there is
such a freedom for software development, the applications are almost equally broad. I believe
the freedom from the tether of the mouse and keyboard are part of the fundamental quality to
this technology, how that freedom is used is up to the developer.
A Budget and Suggested Purchases.
There are a varying array of materials that can be used for this project. There are the necessary
items along with the options for low-end to high-end technology. I’ve also included a couple of
alternative materials which would lead toward a better multi-touch experience.
Essential components
1/4” Acrylic
Flame Polishing
Framing (wood or metal)
Screws and Braces
IR LEDs + Resistors
Power Supply
Mylar
Copper Two-Wire
$5.50 / sq ft.
$1 per linear foot
$25 - $75
$20
(72) 20° 850nm with Resistors | $38
$10 - $20
$3 - $5 per foot | 36”(w)
24¢ per foot
The following can change dramatically on selection
Camera / Lens / Housing
$200
Optoma EP747 Projector
$1499
Personal Computer
$2000
Other suggested materials
Rear Projection Screen
$15.70 per yard / 55”(w)
Silicone Rubber Sheet
$36 per foot / 36”(w)
Purchasing Resrouces
1/4” Acrylic
IR LEDs + Resistors
Mylar
TAP Plastics www.tapplastics.com
www.electronicpartsforless.com (custom packages)
Arch www.archsupplies.com
Rear Projection Screen
Rosebrand www.rosebrand.com
Silicone Rubber Sheet
Acme Rubber www.acmerubber.com
Online Literature Resources.
NUI Group - http://www.nuigroup.com
Features wiki and a message board with global members
discussing their own multi-touch experiments and projects.
Compliant Surface
http://nuigroup.com/forums/viewthread/6/
Multi-Touch Interaction Research - http://cs.nyu.edu/~jhan/ftirtouch/index.html
Low-cost multi-touch sensing through frustrated total internal reflection by Jeff Han
http://portal.acm.org/citation.cfm?doid=1095034.1095054
Synthesis and control on large scale multi-touch sensing displays by Jeff Han
http://portal.acm.org/citation.cfm?id=1142269
Multi-Touch Systems that I have Known and Loved by Bill Buxton
http://www.billbuxton.com/multitouchOverview.html