3D modelling of underwater objects using Photogrammetry

3D modelling of underwater objects using
Photogrammetry*
Petra Helmholz
* Experiments were carried out by Carolyn Martin as part of her master thesis at Curtin University.
Curtin University is a trademark of Curtin University of Technology
CRICOS Provider Code 00301J
Motivation
 Number of Underwater Photogrammetry applications is large,
e.g.
– Inspection of pipelines and man-made-objects.
– Cultural and historical mapping (coral reefs, ship wrecks).
 Underwater Photogrammetry is often expansive (equipment,
process of taken the images) and the image quality is often poor.
 In the last decade, a number of waterproof consumer
cameras e.g. “action cameras” became available
(often for less than $1,000
).
 Many use fisheye lenses which are often not
suitable for geometric accurate photogrammetric
applications
.
21/11/2013
Curtin University is a trademark of Curtin University of Technology
CRICOS Provider Code 00301J
WALIS Forum 2013
Page 2
Image taken from
http://gopro.com/
Image taken from
www.ioncamera.com
Motivation
 New structure-from-motion based
programs provide “everyone” the possibility to
create 3D models based on images
(e.g. VisualSFM, 123D Catch, …)
or
?
 No or little prior knowledge is required to use
these programs.
or
?
 Aim:
– Test of geometric accuracy of a underwater
camera with fisheye lens
(GoPro Hero 3 – black edition)
– Test of Photogrammetric solution against
Structure-From-Motion solution
21/11/2013
Curtin University is a trademark of Curtin University of Technology
CRICOS Provider Code 00301J
WALIS Forum 2013
Images taken from
http://www.cs.cornell.edu
Page 3
Outline
 Background
– Interior orientation parameters and camera calibration
– Relative orientation and scale
– Structure from motion
 Experiments
– Performance above water (using photogrammetric workflow)
– Performance under water (using photogrammetric workflow)
– Accuracy comparison of photogrammetric workflow to structure-frommotion approach
 Conclusions and Outlook
21/11/2013
Curtin University is a trademark of Curtin University of Technology
CRICOS Provider Code 00301J
WALIS Forum 2013
Page 4
Interior orientation parameters
 Interior orientation defines the camera parameters
required for reconstructing the bundle of rays.
 Parameters are:
• The camera’s calibrated focal length (c).
• The position of the principle point (PP) in the
image plane (x’P, y’P).
• Deviations from the ideal sensor model (e.g.
distortion Dr’)  main distortion component is of
radial structure.
 A photogrammetric analysis is possible ONLY when
the interior orientation parameters are known.
21/11/2013
Curtin University is a trademark of Curtin University of Technology
CRICOS Provider Code 00301J
WALIS Forum 2013
Page 5
Sensor plane
Projection
centre O
c
Camera body
Principle Point and Principle Point Offset
 The calibrated focal length c:
• Is the distance of the projection centre
O from the image plane.
Pixel coordinate system (column, row)
Image coordinate system (x, y, (c))
 Principal point:
•
•
y
c
Point of intersection of the perpendicular
from the image perspective centre O with
the image plane.
yp
If the camera is properly adjusted, the
principal point is close to the origin
of the image coordinate system.
xp
 Principle Point offset (xp, yp): is the offset of the principal point
(PP) from the origin of the image coordinate system.
21/11/2013
Curtin University is a trademark of Curtin University of Technology
CRICOS Provider Code 00301J
WALIS Forum 2013
PP
Page 6
x
Lens distortion

Lens distortion Dr’ is present when the actual light ray
in the image space (O’ P’) is not parallel to the
corresponding light ray in object space (OP).
Radial lens distortion: the functional model for radial
lens distortion is an odd-powered polynomial as a
function of radial distance
Dr  k1r 3  k2 r 5  k3r 7
 The coefficients, k1, k2
and k3 are determined
directly (calibration).
O
21/11/2013
Curtin University is a trademark of Curtin University of Technology
CRICOS Provider Code 00301J
WALIS Forum 2013
3
2
1
Image plane

Dr
Image space
P’
Deformed
Image Surface
O
Refracted
Ray Path
(Model and
Actual)
Object
Point
r3
r2
r1
c
r
PP
Page 7
Object space
P
Camera calibration






Suitable target field of object points.
Test field is imaged from several camera stations,
ensuring good ray intersection and filling the image
format.
Convergent image configuration, rather than the
“normal case” traditionally used for aerial photography.
At least 8 images are acquired, each imaging as many
of the test field targets as possible.
The test field is imaged perpendicularly and obliquely
and each image set should have one image rotated by
90° around the optical axis.
Take consistently sharp images (aperture setting, fast
shutter speed, low ISO).
21/11/2013
Curtin University is a trademark of Curtin University of Technology
CRICOS Provider Code 00301J
WALIS Forum 2013
Page 8
test field
Camera positions
Examples for coded
targets (Image
taken from
http://www.iwitnessphoto.com/ )
Orientation Procedures
GCP (X, Y, Z)
≥3
≥3
Adopted from Luhmann et al, 2006 (p. 202)
≥3
≥3
Space resection
Bundle adjustment
Relative
orientation
nl
1
2
Exterior
Orientation
≥2
21/11/2013
WALIS Forum 2013
Absolute
orientation
nN
≥2
Space intersection
Curtin University is a trademark of Curtin University of Technology
CRICOS Provider Code 00301J
≥5
Image points (x, y)
nP
1
New points (X, Y, Z)
Page 9
Video
Structure From Motion
 Simultaneously estimating 3D geometry (structure) and camera position
(motion) (Ullman, 1979).
 Solves the triangulation (intersection), two-frame structure from motion
(relative orientation) & bundle adjustment using the projective geometry.
 Approach mainly used in the Computer Vision
Building Rome in a Day1
community to process big unorganised and
unstructured image datasets.
 For instance, “Building Rome in a Day” project1
 Images taken from Google and Flickr
 Easy to use and looks good, but
what is the geometric accuracy?
1
Agarwal, S., Furukawa, Y. , Snavely, N., Simon, I., Curless, B., Seitz, S.M., Szeliski, R., 2011. Building
Rome in a Day. In Communications of the ACM, Vol. 54, No. 10, Pages 105-112, October 2011.
21/11/2013
Curtin University is a trademark of Curtin University of Technology
CRICOS Provider Code 00301J
WALIS Forum 2013
Page 10
Equipment
 Cameras: Nikon D80 (SLR used in Close Range
Photogrammetry) and GoPro Hero black edition
(consumer camera with fisheye lens).
 Calibration field (box) with coded targets
(above water).
 Calibration frame with reflecting
Calibration frame (underwater)
targets (underwater).
 Software: iWitnessPro (Photogrammtric), VisualSfM (StructureFrom-Motion).
Images taken from Carolyn Martin
21/11/2013
Curtin University is a trademark of Curtin University of Technology
CRICOS Provider Code 00301J
WALIS Forum 2013
Page 11
Calibration box (above water)
Camera calibration above water
 Comparison of the parameter of the interior
orientation from the Nikon and GoPro camera
using the calibration field with coded targets.
 10 images per camera with one rotated image.
 Scale was introduced using observed
distances between reflective targets.
 Both datasets were transformed in the same
local coordinate system defined by 3 reference
points.
 Reflective targets could get pick up
automatically with Photogrammetric software
in the images taken with the Nikon SLR but
not with the GoPro (e.g. motion blur).
21/11/2013
Curtin University is a trademark of Curtin University of Technology
CRICOS Provider Code 00301J
WALIS Forum 2013
Image taken from Carolyn Martin
Page 12
Calibration box (above water)
Camera calibration above water
Nikon D80
GoPro
c
17.943 mm
2.755 mm
xP
-0.016 mm
-0.070 mm
yP
-0.115 mm
0.109 mm
k1
5.488∙10-4
4.408∙10-2
k2
-6.109∙10-7
2.166∙10-3
k3
5.188∙10-10
-6.256∙10-5
Quality index
1:2,100
1:1,900
Accuracy of
referencing
0.63 pixels
1.77 pixels
No. of ref. points
88
82
21/11/2013
Curtin University is a trademark of Curtin University of Technology
CRICOS Provider Code 00301J
WALIS Forum 2013
Radial
Distortion
Grid
Nikon D80:
Two orders of
a magnitude
difference.
Radial
Distortion
Grid
GoPro:
Page 13
Accuracy check above water
 Residuals between Nikon (reference) and GoPro.
Residuals (scale factor 30)
Magnitude of residuals [mm]
average
2.19
max
4.05
z
Standard deviations [mm]
X
1.40
Y
1.35
Z
1.60
21/11/2013
Curtin University is a trademark of Curtin University of Technology
CRICOS Provider Code 00301J
y
WALIS Forum 2013
x
Page 14
Camera calibration underwater
 Use of only the GoPro Camera.
 Reference is given by reference frame.
 Experiment was performed in a water tank at
Curtin University (thanks to the Centre for
Marine Science and Technology (CMST)).
 Lots of motion blur during the images are taken
 only a small subset of images was useable
(6 images).
 Motion blur, low light
and low visibility is in
general an issue in
Underwater
Photogrammetry.
21/11/2013
Curtin University is a trademark of Curtin University of Technology
CRICOS Provider Code 00301J
WALIS Forum 2013
Calibration frame (underwater)
Images taken from Carolyn Martin
Page 15
Camera calibration underwater
GoPro camera
Above water
Under water
c
2.755 mm
3.607 mm
xP
-0.070 mm
-0.030 mm
yP
0.109 mm
0.121 mm
k1
4.408∙10-2
1.845∙10-2
k2
2.166∙
-1.550∙
k3
-6.256∙10-5
3.747∙10-6
Quality index
1:1,900
1:2,400
Accuracy of
referencing
1.77 pixels
0.65 pixels
No. of ref. points
82
34
21/11/2013
Curtin University is a trademark of Curtin University of Technology
CRICOS Provider Code 00301J
10-3
WALIS Forum 2013
10-3
Distortion by
fisheye lens is
smaller.
Similar to Nikon
in previous test
(0.63 pixels).
Smaller dataset.
Page 16
Radial Distortion Grid GoPro
(above water and below water)
Accuracy check underwater
 Residuals between calibration frame (reference) and GoPro
Magnitude of residuals [mm]
Residuals (scale factor 30)
average 1.36 instead of 2.19 mm
max
3.09 instead of 4.05 mm
Standard deviations [mm]
X
1.24 instead of 1.40 mm
Y
0.70 instead of 1.35 mm
Z
1.17 instead of 1.60 mm
z
y
21/11/2013
Curtin University is a trademark of Curtin University of Technology
CRICOS Provider Code 00301J
WALIS Forum 2013
x
Page 17
Comparison to the Structure-from-motion app.
 Software output is a point cloud, an orthoimage, a textured model and a video file.
 Observations in the images are not possible
(no space intersection).
 Points of interest could be not extracted in the
point cloud or the mesh.
Extracted 3D Point Cloud
Back frame
Front frame
Ortho-image
Data taken from Carolyn Martin.
21/11/2013
Curtin University is a trademark of Curtin University of Technology
CRICOS Provider Code 00301J
WALIS Forum 2013
Page 18
Conclusions
 Fisheye effect above water has a very strong
influence and leads to high radial distortions (e.g.
epipolar line is actually a curved line).
 The fisheye effect decreases by underwater
application (distortion pattern and geometric
accuracy improves).
 Structure-from-motion can often achieve good
models but struggles with difficult scenes like in
this extreme example.
 Further investigations in the geometric accuracy
are necessary.
Curved epipolar line
21/11/2013
Curtin University is a trademark of Curtin University of Technology
CRICOS Provider Code 00301J
WALIS Forum 2013
Page 19
Outlook
 In 2008 HMAS Sydney and HSK Kormoran was found
in 2,500 m depth off Geralton’s coastline by the Finding
Sydney Foundation.
 Images were taken without the application of 3D
modelling but some 3D models could be extracted from
the dataset (Hollick et al., 2013).
HSK Kormoran foredeck
2D Photographs © 2008 Australian War Memorial via The Finding Sydney Foundation.
21/11/2013
Curtin University is a trademark of Curtin University of Technology
CRICOS Provider Code 00301J
WALIS Forum 2013
Page 20
Video
Outlook
 Revisit of HMAS Sydney is scheduled for 2014/2015.
 Equipment for the dive (ROV and cameras) is available.
 Goal: to create a 3D model of the ship wreck as complete as possible with a
medium to good geometric accuracy.
 Team at Curtin working on existing data set and on the revisit plans (Andrew
Hutchison, Andrew Woods, Petra Helmholz).  Models will be used in
exhibitions and also for further analysis.
HSK Kormoran foredeck
Image taken from Hollick et al. (2013)
21/11/2013
Curtin University is a trademark of Curtin University of Technology
CRICOS Provider Code 00301J
WALIS Forum 2013
Page 21