View presentation

3D Handwriting Analysis
A. Razdan, J. Femiani, J. Rowe
Partnership for Research in Spatial Modeling
(PRISM)
Dr. Anshuman Razdan
Director
([email protected])
Parsing the OCR Problem
•
•
•
•
8/2/2017
Preprocessing and Image enhancement
Pen Stroke Creation
Character recognition
Word recognition
2
Image Enhancement
• Preprocessing includes enhancing and refining the
raw image.
• Identifying and extracting blurred, stained, faded,
bled through, or transferred characters, etc.
• New PRISM method specifically identifies and
analyzes linear structures (line strokes).
• This technique works in both 3D (CT, MRI) and 2D
(images) domains.
8/2/2017
3
Image Refinement
•
•
•
•
8/2/2017
1D and 2D function models based on the 3 observed shape
characteristics have been developed, and enhanced images are
derived from their second derivatives.
A two-stage algorithm is developed to extract line and net
patterns. Line and net patterns are first enhanced and then
extracted by applying threshold value.
Line and net patterns in a noisy environment exist in many
imaging technologies
Examples: Roads and rivers in satellite photos, curves in finger
prints, blood vessels in CT angiography
4
Enhancement & Thresholding
Original image
8/2/2017
Enhanced image
Line extraction by
thresholding
5
Spanish Manuscript Example
8/2/2017
6
Why 3D Analysis?
8/2/2017
7
Flat Land: A Romance of Many Dimensions
• You have to view the problem in at least one dimension
higher than the data to get a sense of it(Flatland: A Romance
of Many Dimensions: by Edwin A. Abbott, A Square, circa.
1884)
Observer in 2D Land
KING of 1D Land
woman
You are in 3D looking
down at 2D space
8/2/2017
High
Priest
8
An Example
8/2/2017
9
Now I See Now I Don’t
P R I S M K G L M e s h V i e w e r C o n tr o l
C : \ R a z d a n D a t a \ P r i s m \ K D I \ P re s e n ta ti o n s / tu b _ m e s h _ c o n n e c te d . k g l
8/2/2017
10
Flat Land Conclusion
• 1D (line) embed in 2D space (paper surface)
• 2D (images) embed in 3D space (like this
room)
• 3D (objects) embedded in 4D or 5D space ….
• Given this argument, using 3D space for
understanding 2D images makes sense….
8/2/2017
11
3D Pen Traces
8/2/2017
12
3D Pen Trace Recreation
• Concept of raising or embedding 2D image in 3D
space a.k.a Flat Land.
• Understanding ink flow and information embedded in
the pen strokes
• Theory of Volume Modeling and Iso-surface
Extraction
8/2/2017
13
Chain Codes or Pen Traces
• For any character
matching/recognition
algorithm to work efficiently
it needs to unravel the
stroking of the pen.
• This means figuring out the
chain code. Since it is not
available in 2D bitmap we
do it using 3D.
8/2/2017
14
Pen Stroking
• Pressure is applied to via the pen and is different in
upstrokes and down strokes and also angle of writing.
• There is flow of ink from the pen to the paper.
Crossovers result in darker images
8/2/2017
15
How 2D is raised to 3D
• A transfer function is applied which converts
intensity at each pixel into a height function and also
a density function
• Results in Volumetric data same as CT or MRI
H(i,j) = F(x,y, I(x,y))
D(i,j,k) = I(x,y)
Vol Func(x,y,H(i,j)) = D(I(x,y))
2D Image
Transformed into 3D
8/2/2017
16
Marching Cubes
• Marching cubes is used for making 3D surfaces from
volumetric data such as MRI, CAT scan, etc.
8/2/2017
17
MC: Thresholding
•
8/2/2017
Explanation of how Marching Cubes uses predefined
triangulations for each cube to form a whole mesh.
18
Volume Blurring
• Start with Volume Function (V) on raw image (left image)
• Apply Marching Cubes on V (middle image)
• Create V’ = GnV (Blurring filter applied n times and then MC to
create right image). Gn is the secret sauce.
8/2/2017
19
Modern Writing
8/2/2017
20
Demo of Current Implementation
8/2/2017
21
Curve Shape Measures and
Matching for Character Recognition
The Problem
• Given two curves X1 and X2, one can ask two
distinct questions:
– Curve matching i.e.
• Is X1 = X2 ?
• Or one a subset of the other curve
• Or how similar are the two curves?
– Curve alignment i.e.
• What is the rotation and translation required to align one
curve with the other?
8/2/2017
23
Curve Matching Applied to Chars (Demo)
8/2/2017
24
Conclusions
• Novel method to unravel strokes, characters and letterforms in
complex handwritten documents.
• Segments by Region/Row irrespective of scale, orientation, or
position.
• Geometry based curve matching technique for character
recognition (dictionary generation, text recognition, and
translation)
• Language independence
• Doesn’t need expensive scanning equipment (we paid $24.99).
• Can be combined with existing technologies.
• Provisional Patent filed in April 2003. Full patent filing spring
2004.
8/2/2017
25
Partial Match
8/2/2017
26
Best Match
8/2/2017
27
Weaknesses
• Requires continuous tone original source (can not
address single bit image i.e. FAX).
• Can be computationally expensive for certain
applications such as forgery but the technology is built
to take advantage of parallelization.
8/2/2017
28
Opportunities
• Extend concept of volumes to other applications
–
–
–
–
Forensics (Offline comparisons)
Biometrics (Online authentication – wacom demo)
Forgery detection
Number extraction from noisy background (Currencies)
• Opportunities for derivative patents
8/2/2017
29
Gaps
• Need to combine power of Stroke extraction and curve
matching with traditional HMM and other statistical
methods or commercial engines.
• Man power/expertise required
– AI/Statistics/traditional char recognition expert to create
powerful hybrid engine
– Language specific expert/paleographer
• Requires productization and field testing.
8/2/2017
30
Threats
• Competition by 2D solutions and existing
technologies.
• Lack of awareness of the capabilities of 3D analytical
tools in OCR world.
– Geometry solution in a world seeped in statistical methods.
• Establishing validity of the 2D - 3D conversion
algorithm
8/2/2017
31
Discussion and Q/A
8/2/2017
32
Appendix
8/2/2017
33
PRISM Infrastructure
• Two labs on campus
– 0ne moving to bigger space in BY – downtown Tempe.
– Additional 8000 sq ft slated for a new project (Decision
Theatre) in downtown Tempe.
•
•
•
•
24 proc SGI, 20+ workstations (Unix, PC and Linux)
Four 3D Laser scanners for inanimate objects
3D face scanner (recent acquisition)
2 Rapid Prototyping machines
8/2/2017
34
Image Refinement
• Biomedical Examples: White matter in brain MRI
scans, cell spindle fibers, membranes in laser confocal
microscopic data.
Fungus membrane
Brain MRI Scan
8/2/2017
Mouse egg
35
Image Refinement
• Blood Vessel
3 characteristics
(Chaudhuri et al)
1. Piecewise linear
segments
2. Cross section as a
Gaussian function
3. Relatively constant
width
8/2/2017
36
2D Line Model
Blood Vessel
(x,y)
v  (cos  , sin  )

  x cos  y sin  
F ( x, y )  exp  
2
2

2
8/2/2017




37
2D Case: 2nd Derivatives
 ( x cos  y sin  )2 
  C  N ( x, y )
F ( x, y )  exp  
2
2



Fxy ( x, y )  Fyx ( x, y )  

cos sin 
4
Fxx ( x, y )  
cos2 
2
cos sin 
2
C: constant, N: noise
 ( x cos  y sin  ) 2 

exp  
2
2



 ( x cos   y sin  ) 2 
  N xy ( x, y )
( x cos  y sin  ) exp  
2
2



2
 ( x cos  y sin  ) 2 

exp  
2
2



cos2 
 ( x cos  y sin  ) 2 
  N xx ( x, y )

( x cos  y sin  ) exp  
2
4
2



 ( x cos   y sin  ) 2 
sin 2 

Fyy ( x, y )  
exp  
2
2
2




8/2/2017
sin 2 
4
2
 ( x cos  y sin  ) 2 
  N yy ( x, y )
( x cos   y sin  ) exp  
2
2



2
38
Enhancement
• Maximal eigenvalue as an enhanced image
 Fxx
Hv  
 Fyx
Fxy  cos 
(  if x cos  y sin    )
Fyy   sin  
 ( x cos  y sin  ) 2   cos2 
 
  2 exp  
2

2

 sin  cos
 ( x cos  y sin  ) 2  cos 
1
 
  2 exp  

2

2

  sin  
1
  ( x, y ) v   2 F ( x, y ) v
1
sin  cos  cos 


sin 2    sin  

Enhanced Image
   2  ( x, y )
F ( x, y )  
0
8/2/2017
if  ( x, y )  0
if  ( x, y )  0
39
Results
8/2/2017
A synthetic image
Crest lines
extraction
Matched filters
Our method
40
Applications of Curve Matching
8/2/2017
41
Distance Between Two Functions
Case 1: f and g continuous over [0,1]
Case 2: f over [0,1] and g over [0,d], d <= 1
Penalty function
8/2/2017
42
Curve Shape Measures
• Shape Measures or Properties
– Curvature (planar)
– Torsion (space curves)
– Total or absolute Curvature
(space)
• Classical Differential geometry
says if the curvatures are
identical then so are the curves
subject to position and rotation
8/2/2017
43
Curve Matching
• Remember
• Writing in terms of
curvatures
• What about partial
match?
• Or the general case
8/2/2017
44
Three Matching Mesaures
8/2/2017
45