Segmentation SIRS talk

Segmentation
Adding semi-automated image
segmentation to AnnoteImage
Todd Detwiler – UW SIG
AnnoteImage
z
z
z
z
Labels assigned to image regions
Regions defined by bounding polygons
Polygons defined by user placed points in
image
Placing points in image is a manual process
z
z
z
Zooming
Scrolling
Clicking
AnnoteImage (“before” demo)
Improving AnnoteImage
z
Why should we improve it?
z
z
z
VSP myocardial regions
Commissure radiological atlas
How should we improve it
z
z
Add semi-automatic edge tracing capability to
allow authors to place points faster
Similar to PhotoShop’s Magnetic Lasso tool
Intelligent Scissors
z
z
z
MORTENSEN, E., AND BARRETT, W. 1995.
Intelligent scissors for image composition.
Proc. ACM Siggraph, 191-198.
Steve Seitz, Computer Vision (CSE576,
Winter 2005).
Intelligent Scissors = LiveWire
Intelligent Scissors
Computing link costs
Computing link costs
Grayscale:
D(link1)=|img(i+1,j)-img(i,j-1)|/sqrt(2)
D(link 0)=|(img(i,j-1) + img(i+1,j-1))/2 - (img(i,j+1) + img(i+1,j+1))/2|/2
D(link2)=|(img(i-1,j)+img(i-1,j-1))/2-(img(i+1,j)+img(i+1,j-1))/2|/2
cost(link)=(maxD-D(link))*length(link)
Computing link costs
RGB:
Intensity derivatives across each channel ( DR(link),DG(link),DB(link) )
D(link) = sqrt(
(DR(link)*DR(link)+DG(link)*DG(link)+DB(link)*DB(link))/3 )
cost(link)=(maxD-D(link))*length(link)
Notes:
•
Lots of symmetry to utilize
•
D(link) formulas can be implemented as 3x3 cross correlation filters
Computing link costs
0
1/4
1/4
0
0
0
0
-1/4
-1/4
Computing link costs
0
1/ 2
0
0
0
−1/ 2
0
0
0
Live-Wire 2D DP Graph Search
z
Modified Dykstra’s algorithm
z
z
z
Compute lowest cost path from seed node to
every node in image
Repeat for every new seed node
Dynamic programming – reuse solutions to
subproblems
Live-Wire 2D DP Graph Search
AnnoteImage (“after” demo)
Calculating polygonal points
z
z
z
Not part of Intelligent Scissors
Want few points where they approximate a
straight line, but more in areas of high
curvature/detail
Some methods tried
z
z
z
z
Angle based measure
Area of triangle based measure
Angle sum based measure
Turn angle based measure
Angle based measure
z
Idea: If the angle between 2 lines, connecting a
point on the path to 2 subsequent points on the
path, exceeds a certain threshold then we should
keep the intermediate point
Angle based measure
z
z
Favors short segments
Unsatisfactory results even after scaling by length
(non-linear)
θ
Area of triangle based
measure
z
z
Idea: If the area of a triangle, formed by 3 points on
the path, exceeds some threshold, then the middle
point should be retained
This got rid of the bias towards short segments, but
still yielded unsatisfactory results (choice to retain
midpoint largely just a function of distance now).
Angle sum based measure
z
Idea: Score each segment by the degree in which it
deviates in direction from some original segment.
When the cumulative deviations exceed some
threshold, place a point on path
2
4
2
1
1
0
0
1
0
-1
2
Angle sum based measure
z
Problem
2
4
1
1
0
0
2
0
1
0
1
0
0
0
0
4
Turn angle based measure
z
z
Lee, D.J., Antani, S. and Long, L.R., 2003. Similarity
measurement using polygon curve representation
and fourier descriptors for shape-based vertebral
image retrieval. In: Proceedings of
IS&T/SPIEMedical Imaging 2003: Image
Processing, Vol. SPIE 5032, February 2003, San
Diego, CA, pp. 1283–1291.
S. Antani, D.J. Leeb, L.R. Longa, G.R. Thoma:
Evaluation of shape similarity measurement
methods for spine X-ray images in J. Vis. Commun.
Image R. 15 (2004) 285 302.
Turn angle based measure
Paper:
Modified:
K ( s1 , s2 ) =
π − β ( s1 , s2 ) l ( s1 )l ( s2 )
l ( s1 ) + l ( s2 )
Improvements
z
z
z
Use LiveWire to calculate path to seed BOTH
ways
Allow user refinement of how closely polygon
should fit LiveWire path (point reduction)
Priority queue may not be the optimal way of
sorting
z
z
z
Java PriorityQueue provided O(log(n)) for
insertion and constant time removal
Bucket or radix sort could reduce this to O(n)
(not the same n)
Toboggan-based intelligent
scissors
z
z
E. N. Mortensen, Simultaneous Multi-Frame
Subpixel Boundary Definition using TobogganBased Intelligent Scissors for Image and Movie
Editing, Doctoral Dissertation, Department of
Computer Science, Brigham Young University,
Provo, UT, Dec. 2000.
E. N. Mortensen and W. A. Barrett, "TobogganBased Intelligent Scissors with a Four Parameter
Edge Model," in Proc. IEEE: Computer Vision and
Pattern Recognition (CVPR '99), Vol. II, pp. 452458, Fort Collins, CO, June 1999.
Toboggan-based intelligent
scissors
Toboggan-based intelligent
scissors
z
z
Idea: Reduce size of the graph to improve the
speed of the graph search
Method: Graph size reduction accomplished
by a preprocessing step that over-segments
the image
Toboggan-based intelligent
scissors
z
z
Started, not completed
It’s one thing to get the basic idea from
reading a paper. It’s another thing entirely to
implement it.
“In theory, there's no difference between theory
and practice. In practice, there is.”
- Unknown
Multi-scale gradient
Multi-scale gradient
Multi-scale gradient
Sobel filter
Gradient magnitude image
Toboggan segmentation
Toboggan segmentation
Graph creation
z
z
z
Much more complicated
Edges map to the boundary segments
between regions
Nodes exist where 3 or 4 regions meet (note
these are not at a pixel)
Status
z
z
z
z
z
Sobel gradient magnitude image
Segmentation appears to be working
correctly (although algorithm was slightly
modified from the one introduced in paper)
Graph creation implemented, but is most
likely not correct yet
Edge weights not implemented
New graph search not implemented
Conclusion
z
z
z
z
Intelligent scissors (or LiveWire) seems to be
a reasonable approach to providing
segmentation assistance
Only works well for “hard” edges
Original implementation is slow on very large
images
Augmentations are difficult and riddled with
corner cases