Loading and validating Oracle spatial data

OUGF 03.11.2011
Loading and Validating data in Oracle Spatial
Tomi Salmi
Content
•
•
•
•
•
•
•
•
•
Few words about Karttakeskus and presenter
What is Oracle Spatial?
SDO_GEOMETRY and the bits and pieces
Loading GIS data to Oracle
Validating loaded spatial data
Fixing invalid data
Tools available (SQL Developer etc.)
Pitfalls and best practices
Questions?
Tomi Salmi
•
•
•
•
•
Oracle 9i and 10g OCP DBA
10 years experience in Oracle Spatial and PL/SQL
Project Manager
Technical Architect
Database Specialist
All geospatial information
services in one place
• Dedicated technology independent IThouse
• Highly skilled trainer and consultant
• Finland's leading GIS
outsourcing services provider
• More than 90 years a unique expertise
in spatial data
• All combined with strong consumer view
11/4/2011
Karttakeskus
4
Karttakeskus Oy
part of Affecto Group
• Finland's largest only GIS focused
company
• Incorporated 1.1. 2011
• Turnover of 11M €
• Staffed by 80 GIS professionals
• Affecto's 100%-owned subsidiary
• Affecto Group is the Nordic region's
leading EIM solution provider
– Turnover of 114 M €
– and employs 947 the end of 2010
11/4/2011
Karttakeskus
5
What is Oracle Spatial?
• Extension available to Oracle Database Enterprise Edition
• Limited free version ”Oracle Locator” is included in all database
editions
• SDO_GEOMETRY-datatype
• Functions and operators for data queries and data handling
• EPSG Coordinate systems
• Routing and network calculations
• GeoRaster
• Geocoding
• Web services
• 3D data in Oracle 11g
History of Oracle Spatial
SDO_GEOMETRY
• Geometry is saved as SDO_GEOMETRY-object type, where
everything about geometry is in one column
• SDO_GEOMETRY is a object type:
CREATE TYPE SDO_GEOMETRY AS OBJECT
(SDO_GTYPE NUMBER,
SDO_SRID NUMBER,
SDO_POINT SDO_POINT_TYPE,
SDO_ELEM_INFO MDSYS.SDO_ELEM_INFO_ARRAY,
SDO_ORDINATES MDSYS.SDO_ORDINATE_ARRAY
);
• SDO_Geometry object has few methods to access its attributes
Metadata for Spatial columns
• MDSYS.SDO_GEOM_METADATA_TABLE
– Table owned by MDSYS-users. Contains metadata of ALL spatial
columns of database’s tables
– No updates to this table
• USER_SDO_GEOM_METADATA
– View to user’s tables containing spatial data column
– Only view to be used to modify spatial metadata
– Doesn’t get updated automatically!!!!!
• ALL_SDO_GEOM_METADATA
– View containg info for all tables user has atleast select priviledges
Coordinate systems
• Oracle Spatial has full support for geodetic and projected coordinate
systems and transformations between any two of them
• Oracle 10g R2 included support for EPSG and corrections for
transformations
• In Oracle, the coordinate system for spatial data is chosen using
approriate SDO_SRID-value (if NULL, planar coordinate system is used)
• Table MDSYS.CS_SRS contains information about all available
coordinate systems
– User can always define own coordinate systems (there are limitations)
• MDSYS.SDO_DATUMS has information about datums
• MDSYS.SDO_ELLIPSOIDS has information about ellipsoids
• MDSYS.SDO_PROJECTIONS has information about
projections
Tools to load spatial Data into Oracle Database
• Oracle Spatial doesn’t provide great tools to load spatial data
(and rasters) into database
• Many professional GIS-software supports loading data into
Oracle Spatial
• Oracle Spatial includes free software called shp2sdo ”Shape to
SDO”
– It can be downloaded from Oracle’s web-pages
– Tool processes ESRI Shapefiles to text files, which can be loaded
into Oracle spatial using SQL*Loader
• But now we have .......
SQL Developer + GeoRaptor
Spatial indexes
• Like normal queries in Oracle, spatial queries need indexes to be fast
• Most of the spatial functions will not work without spatial index
• CREATE INDEX TOMI_SPX on TOMI(GEOLOC)
INDEXTYPE IS MDSYS.SPATIAL_INDEX;
• Spatial metadata must be set before you can create a spatial index
• If you want to change coordinate system for the geometries, you have
to drop/create the index
• Creation of the spatial index takes lot’s of time and space
• Oracle has two types of spatial indexes: Quadtree and R-tree
Spatial indexes
• Oracle9i included new spatial index option R-tree
– In R-treeindex, MBR (Minimum Bounding Rectangle) is calculated
for every geometry
– Index has a tree structure where parent nodes MBR is calculated
from child nodes MBR
– Creating R-tree indexes is easy, because you don’t have to define
any parameters about index structure (like in quadtree)
– Indexing 3D geometries is possible
Spatial indexes
• Oracle’s spatial query is divided into two phases:
– Primary filter will use spatial index to gather all candidates for the
query’s result
– Secundary filter will make detailed spatial calculations to
determine the final resultset (slow)
Loading and validating Oracle spatial data
• Oracle is really picky about spatial data:
– Data must be validated and fixed to be topologically correct
Otherwise spatial operations and queries can
return incorrect data/results
• Most important aspect of the validation is the tolerance value stored in
spatial metadata table
– Using this tolerance, Oracle decides if two adjacent coordinate points
are same/different
Loading and validating Oracle spatial data
• Picture has to objects A and B and distance between them
is d.
• Tolerance saved in metadata-table for geometry column
determines what is a relation between these two objects:
– If distance d is less than tolerance, the those objects are adjacent
(B resides in A’s edge, e.g objects ”touch”)
– If distance d is equal or greater
than tolerance, Oracle determines
these those objects are disjoint
and doesn’t ”touch” each other.
Loading and validating Oracle spatial data
• Same principle for the tolerance is used when validating
relations between coordinate points of a single geometry
object:
– Is the duplicate points
– Does parts of polygon overlap itself (not allowed, different polygon
parts can only touch each other)
• Tolerance value should be chosen to correlate the
accuracy of the data.
• For example, if precision of the coordinate values are in
one decimal (0.1 , 0.2), tolerance should be half of this
-> 0.05
Loading and validating Oracle spatial data
• SDO_GEOM.VALIDATE_GEOMETRY_WITH_CONTEXT(
geometry IN SDO_GEOMETRY,
tolerance IN NUMBER (tai diminfo IN SDO_DIM_ARRAY)
) RETURN VARCHAR2;
• SDO_GEOM.VALIDATE_LAYER_WITH_CONTEXT(
table_name
IN VARCHAR2,
column_name IN VARCHAR2,
result_table
IN VARCHAR2
[, commit_interval IN NUMBER ] );
• These are the most often encountered ORA-errors and their
descriptions:
– ORA-13356 - Adjacent repeated points in a geometry are redundant
– ORA-13349 - Polygon boundary crosses itself (polygoni leikkaa itseään)
– ORA-13367 - Wrong rotation for interior/exterior rings
How to fix invalid geometries?
• Oracle provides few tools to correct invalid geometries
– Some of them can change the shape of the geometry
• Area etc. can change!!!!
• SDO_MIGRATE.TO_CURRENT
– Procedure corrects certain errors for all geometries after loading data
into database
– This procedure has a long history. Oracle uses it when migrating from
really old Oracle versions (8i mostly)
– Corrects orientation wrong SDO_GTYPE, SRID etc. values
– Doesn’t actually correct ”digitization” of the geometry
– It’s good practice to use this procedure after loading spatial data from
files
How to fix invalid geometries?
• SDO_UTIL.REMOVE_DUPLICATE_VERTICES
– This function removes duplicate (redundant) vertices (points, lines)
from a geometry
– Returns corrected geometry
• SDO_UTIL.RECTIFY_GEOMETRY
– This function ”tries to fix it all”
• Duplicate vertices
• Polygon boundary intersecting itself
• Incorrect orientation of exterior or interior rings (or both) of a polygon
– Returns corrected geometry
• These two functions can be found from Oracle10g upwards
• If you have 8i/9i you can try to use SDO_UNION-function
DEMO
1. loading SHP-file into oracle with
SQL Developer
2. Enter/update metadata
3. Validating data and visualizing
found errors
4. Correcting found errors
Pitfalls and best practices
• Oracle’s polygon node orientation < > OGC standard
– Always validate/migrate data after loading it to Oracle
• SDO_UNION in special cases when loading SHP-files
– GIS-softwares usually handle this correctly
– WFS softwares can return geometry from Oracle ”as is”
• GIS software’s equal-function doesn’t work
• If you mass insert/update spatial data
– Drop/create spatial index
– Remember validation before creating the index
– Remember to check that metadatas boundaries are valid (all
geometries actually are inside it)
Pitfalls and best practices
• If you have HUGE amount of data in same table
– Partitioning (spatial indexes support this now)
– Order data spatially (or with other key) inside the table
If you got interested ....
• http://www.spatialdbadvisor.com
– Really good webpage by independent Spatial Consultant
• Pro Oracle Spatial for Oracle Database 11g
– Excellent book about Oracle Spatial.
• http://forums.oracle.com/forums/forum.jspa?forumID=76
– OTN Forum’s Spatial-section
• https://support.oracle.com
– Unfortunately Oracle Spatial has bugs ....
• RTFM
– Oracle’s own Spatial manuals are really good
Thank You !!!!
Questions ?
[email protected]