Computer vision

Why Computer Vision became a
consumer reality
For Next-Generation Wearable and Mobile Applications
CDNLive
August 13, 2015
Yipeng Liu
Technical Marketing Director
Outline
• Few definition first
– Camera pipeline for computer vision
– Computer vision: past and present
• Computer Vision in mobile, wearable, tablet – all consumer
devices
– Driver: Killer use cases, Power, performance
• Example killer use cases
• Example: consumer devices with power challenges
• Choosing right solution for imaging and vision in consumer
market
• Cadence Imaging and Vision Solutions
2
© 2015 Cadence Design Systems, Inc. All rights reserved.
Camera pipeline: For today and tomorrow
Sensor
Pre
Processing
Bayer Domain
Type of
Processing
Lens Shading
Defect Correction
HDR
Phase Detection AF
HW Pipeline
Bayer to RGB
Demosaic
Color matrix
Image processing
3
© 2015 Cadence Design Systems, Inc. All rights reserved.
Post
Processing
(YUV)
2D/3D Noise
Reduction
Image
Stabilization
HDR/WDR
Super resolution
Image/Video
Analysis
Face Detection
People Detection
Object Detection
Gesture Detection
Motion Detection
Vision
processing
Computer Vision History
• Computer Vision Definition:
– Computer vision is a field that includes methods for
acquiring, processing, analyzing, and understanding images
duplicating the abilities of human vision by electronically perceiving
and understanding an image
• Brief History
– Computer Vision field has been around since 1950’s
– First in academia: pattern recognition with large scale computers
– Followed by use in manufacturing line
• Now: Use of computer vision in mobile & wearable
– But why?
4
© 2015 Cadence Design Systems, Inc. All rights reserved.
Computer Vision in today’s consumer devices
Driven by three major factors
Low
Energy
5
Longer battery life
High
Performance
Killer Use
Cases
Minimum performance:
Resolution
Frame rate
Simplify usability
Improve safety & security
Increase efficiency
© 2015 Cadence Design Systems, Inc. All rights reserved.
Killer use cases: always on/authentication/video
chat/gesture/sports photography
• Always on: wakeup on face detection and voice trigger
• Authentication
− Face recognition and voice recognition, iris detection (biometrics)
• Quick video chat: 4K video chat?
• Selfies: How about a 3D selfies?
• Gesture control
Source: Sensory
Gesture Control in Automotive
Gesture Control in Gaming &
Home Control
• Sports photography
•
•
•
•
6
HDR
Video stabilizer,
High frame rate (60,120 to 240)
Continuous capture
Source: eyeSight
Video Stabilizer
© 2015 Cadence Design Systems, Inc. All rights reserved worldwide. Cadence and the Cadence logo are trademarks of Cadence
© 2015Systems,
CadenceInc.
Design
Systems,
Inc. Alland
rights
reserved.
Design
in the
United States
other
countries. All other trademarks are the property of their respective owners
and are not affiliated with Cadence.
HDR
Killer use cases: 3D Capture
Needs Stereo Solution
Functional Measurements
source: PercepTonic http://perceptonic.com/case-studies.html
•
•
Face capture & print (3D selfie Printing)
Gesture recognition
•
Functional measurement
• DIY
• Does the furniture fit in my room?
• On-line retail – scan to
search/fit/sale
© 2015 Cadence Design Systems, Inc. All rights reserved worldwide. Cadence and the Cadence logo are trademarks of Cadence Design Systems,
Inc. in the United States and other countries. All other trademarks are the property of their respective owners and are not affiliated with Cadence.
7
© 2015 Cadence Design Systems, Inc. All rights reserved.
Real-life example: Wearable Devices
Google Glass Power Consumption (W)
3.32
2.93
3
• Face detection running
at VGA (640x480)
resolution takes: 3.3W
total power
• Battery runs out in
38 minutes
• While running video
chat, temperature
runs as high
as 52°C
• Android Ware uses
1.2GHz, dual-core
Snapdragon (something
that would have been in
a flagship phone not all
that long ago)
2
W
Apple Watch & Android
Ware
• Apple Watch uses
a relatively powerful
processor and highquality screen, both of
which contribute to
significant power drain.
3 hours of talk time1
2.96
2.36
2.5
Google Glass
2.93
1.5
1.26
1
0.5
0
Phone
Calls
Camera
Preview
call back
Image
Capture
Video Video Chat Face
Capture
Detection
Source: Robert Likam Wa,Aaron Carrol: Draining Our Glass: An Energy and
Head Characterization
 Major battery drain with camera
use for video chat, video quickie
 CPU + GPU are not suitable for
imaging and vision applications on
wearable products
1
8
www.techradr.com
© 2015 Cadence Design Systems, Inc. All rights reserved.
Imaging/vision: performance vs. flexibility
Performance/mW
Hardware
Pipeline
Software
on
GPU
Development
(years)
9
Software on
Imaging/Vision
DSP
© 2015 Cadence Design Systems, Inc. All rights reserved.
Software
on
CPU
Development
(months)
Cadence Tensilica® IVP Family
© 2015 Cadence Design Systems, Inc. All rights reserved worldwide. Cadence, the Cadence logo, and
10
are registered
trademarks
Cadence
Design
Systems.
©Tensilica
2015 Cadence
Design
Systems,ofInc.
All rights
reserved.
Tensilica Products
Providing the most efficient processors in most markets
IoT, Mobile Phones, Storage/SSD, Networking, Video, Security, Cameras, Watches, Printers, others...
Control
Custom
Control
HiFi
ConnX
Vision
Fusion
Custom ISA
Embedded
Audio / Voice /
Speech
Communications
Computer Vision
/ Imaging
General Purpose
Applicationspecific
• Narrow to wide
band wireless
• LTE/LTE-A,
WiFi,
SmartGrid
• Infrastructure
& Terminals
• Image
Processing &
Analytics
• Scalable DSP
• Always-alert
• Sensor
processing
• Audio / Voice /
Speech
• Comm’s
• Advanced
compiler and
development
tools support
• Energy & area
efficient
Thousands
of designs
11
Wide range of DSPs
• Encode +
Decode
• Voice Trigger
• Noise
Reduction
• PostProcessing
• Video Pre-Post
Processing
Tensilica Optimization Platform
Common Architecture, Development Tools, 3rd Party Ecosystem
© 2015 Cadence Design Systems, Inc. All rights reserved.
• Low Energy
• High
Performance
• Application
datatypes
>2B cores
per year
Tensilica IVP-EP—Image and vision processing
work-horse
I RAM
192x 8b vector
96x16b vector
32x 32b vector
I Cache
IVP-EP Vision
Core
AXI
AXI
•
IVP optimized Xtensa® tools for
maximum productivity in C/C++ and
system modeling
•
Customizable for even higher
performance
•
Multi Processor Ready
12
Tensilica® Imaging and Video
Processor-Enhanced Performance (IVPEP)—newest member of IVP family
•
Massive throughput for demanding
imaging and vision workloads: hundreds
of pixel ops per cycle per core
•
Power-managed pipeline for lowest
energy per operation
Data
RAM
© 2015 Cadence Design Systems, Inc. All rights reserved.
Micro DMA
IVP-EP Vision
Core
Data
RAM
4x 32b scalar
192x 8b vector
96x16b vector
Data
RAM
Micro DMA
I RAM
32x 32b vector
I Cache
4x 32b scalar
Data
RAM
•
DSP value add: low power over host CPU,
host CPU + GPU
CPU/GPU Offload Energy Comparison: Noise Reduction
2000
mJ per Frame
1600
1200
800
400
0
Host CPU (4 cores)
Host CPU (4 cores) + 3-pipe
GPU (4-core)
IVP
>25X
Energy Efficient
compare to CPU
13
© 2015 Cadence Design Systems, Inc. All rights reserved.
IVP-EP value add: rich heritage, fast time to
market
Developed on rich
Xtensa® heritage
• >15 year history of Xtensa
architecture
• Billions of cores in
production
14
Seamless
development
environment
• Eclipse-based IDE GUI
• Xtensa C/C++ (XCC)
Compiler with autovectorization
• Vision library
© 2015 Cadence Design Systems, Inc. All rights reserved.
Complete package
for integration
• RTL, EDA script
• Cycle accurate instruction
set model
• Fast function simulator
(TurboXIM)
• XTSC System C system
modeling
• RTL ready for FPGA
• FPGA platform
Constantly changing optics, sensors, and algorithms
need highly programmable/flexible engine
OpenCV and
OpenVX Library
Imaging/Vision
Kernels
Imaging/Vision
Application Support
15
• 800 library functions optimized for IVP-EP
• Planned OpenVX support
• Filters: Sobel, median, Gaussian filters
• Object detection: ORB, SURF, Harris Corner, HOG
• Optical flow: Lucas-Kanade
•
•
•
•
HDR, video WDR
Image stabilization
Face/people detection
Face recognition
© 2015 Cadence Design Systems, Inc. All rights reserved worldwide. Cadence and the Cadence logo are trademarks of Cadence
Design Systems, Inc. in the United States and other countries. All other trademarks are the property of their respective owners and
© 2015
Cadencewith
Design
Systems,
All rights
reserved.for informational purposes only and does not represent a commitment to
are
not affiliated
Cadence.
ThisInc.
roadmap
is provided
deliver any of the features or functionality discussed in the materials.
Morpho video WDR for Cadence IVP-EP
• Morpho video WDR: single image-based luminance
enhancement technology for video
• Automatic local brightness adjustment for mobile video
experience
• With IVP-EP, total cycle count reduced to less
than 1/16th
• One of the component algorithms accelerated
to 1/73rd in cycle count
Morpho Video WDR: 1080P30FPS
282MHz
IVP-EP
CPU
-300
2.2 GHz
200
700
1200
1700
2200
MHz
© 2015 Cadence Design Systems, Inc. All rights reserved worldwide. Cadence and the Cadence logo are trademarks of Cadence Design Systems,
Inc. in the United States and other countries. All other trademarks are the property of their respective owners and are not affiliated with Cadence.
16
© 2015 Cadence Design Systems, Inc. All rights reserved.
Summary
Why?
Requirements
Cadence®
Tensilica®
IVP-EP
17
• Killer use cases with lower power consumption and high
performance (in terms of frame rate & resolution) presents
ideal conditions for Computer Vision in Consumer devices
• Imaging and vision functions require highly programmable
solutions as there are no standard algorithms and algorithms
are evolving
• The optimum solution for the problem
• Low power, high performance, highly programmable
• Complete solution with DSP core, development tools, software
library, FPGA platform
© 2015 Cadence Design Systems, Inc. All rights reserved.
© 2015 Cadence Design Systems, Inc. All rights reserved worldwide. Cadence, the Cadence logo, Tensilica, and Xtensa are registered trademarks of Cadence Design Systems, Inc. in
the United States and other countries. All other trademarks are the property of their respective owners.