How I learned to love XML! (FME Certified Professional, Peter
Transcription
How I learned to love XML! (FME Certified Professional, Peter
How I learned to ‘love’ XML Peter Laulund National Survey and Cadastre Agenda • KMS and INSPIRE • About XML/GML • Writing XML with FME – Templates – Schema mapping, semantic – Schema mapping, geometry – Workflow – design INSPIRE • INSPIRE is a European initiative to create a common SDI • Specific datasets has to be available in a harmonized way • KMS is the National contact point • KMS has five Annex 1 datasets • I will talk about XML not INSPIRE KMS datasets • • • • • • Transport Networks Hydrography Cadastral Parcels Geographical Names Administrative Units Documentation - http://inspire.jrc.ec.europa.eu Warning: getting through the PDF and the related XSD(s) is a tough read ! Inspire-foss FME and XML XML • XML: eXtensible Markup Language – Syntax used to describe data • GML: Geography Markup Language – XML dialect for describing geography – GML 2, GML 3.1.1, GMLSF, GML 3.2.1 • XSD: XML Schema Definition – XML dialect for describing the contend of xml files XML - example <?xml version="1.0" encoding="UTF-8"?> <!-- oprettet af Pel, kms, 3. august 2012--> <venner xmlns:p="http://www.kms.dk/xmlschmas" xmlns:d="http://www.kms.dk/xmlschmas"> <p:person id="345"> <p:navn>Peter Laulund</p:navn> <p:adresse>Sognegårds alle 54</p:adresse> <p:født> <d:dato> <d:dag>3</d:dag> <d:måned>maj</d:måned> <d:år>1957</d:år> <d:klokken/> </d:dato> </p:født> <p:telefon type="fastnet">+45 36499408</p:telefon> <p:telefon type="mobil">+45 26273031</p:telefon> <p:giftMed href="#445"/> <p:arbjedsgiver/> </p:person> </venner> FME and XML • • • • • • FME reads and writes XML/GML Converts geometry to gml XMLSampleGenerator XMLTemplater XMLValidator XMLFormater XMLTemplater • An XML template is an XML document with XQuery functions <gn:text>{fme:get-attribute("name")}</gn:text> <au:geometry> {fme:get-xml-attribute("gml_geom")} </au:geometry> {fme:get-xml-list-attribute("level{}.xml")} <gml:featureMembers> {fme:process-features("FEATURE")} </gml:featureMembers> XMLTemplater • The document may be loaded from – an attribute – a file – entered into the transformer • We use a file that is loaded into an attribute Templates • Use the XMLSampelGenerator to create the template • Edit the template in a text editor – Delete – Add XQuery function calls • Use XMLValidator to evaluate the result • Use XMLFormater to make it look pretty Writing INSPIRE GML Challenges • Five datasets some with more than one feature type • Data for download from ftp don with FME • WFS with Snowflake • WMS with ? • All data in one Oracle database Dataflow in KMS Oracle *.GML Read Write Schema Mapping - semantik Schema Mapping - geometry Transform to xml Sql FME tools Tcl Aggregate SetTraits OGCGeometry XMLTemplate XMLValidate XMLFormater Schema mapping Schema mapping are basic FME functionality • • • • Add or remove attributes Change feature types Alter domain values All our data are in an Oracle database we will therefore use sql for schema mapping Schema mapping F eatureT yp e REGION REGION REGION REGION REGION REGION REGION : : : : : : : : oldAttribute REGIONKODE DAGI_ID FEAT_ID TIMEOF_CRE FEAT_TYPE REGIONNAVN DQ_RESPONS : : : : : : : : n e w A t t r i b ut e nationalCode localId localIdGeom beginLifespanVersion nationalLevelName name sourceOfName F eatureT yp e REGION REGION REGION REGION REGION : : : : : : newAttribute namespace namespaceGeom country gmlTemplate nationalLevel : : : : : : value dk.kms.au dk.kms.au.geom DK AdministrativeUnit 2ndOrder Schema mapping Europavej Primærvej Sekundærvej Anden vigtig vej Større lokalvej Lokalvej Indkørselsvej Anden vej Read Hovedsti Cykelsti langs vej Sti, diverse Trafikvej-Gennemfart Trafikvej-Fordeling Lokalvej-Primær Lokalvej-Sekundær Lokalvej-Tertiær Ikke tildelt ? mainRoad firstClass secondClass thirdClass fourthClass fifthClass sixthClass seventhClass eighthClass ninthClass Write inspireId All features must have an inspireId, it is a complex type made of • A namespace - <country>.<organisation>.<dataset> • Id – the features database id • Version – null, sequence or timestamp <cp:inspireId> <base:Identifier> <base:localId> {fme:get-attribute("localId")} </base:localId> <base:namespace> {fme:get-attribute("namespace")} </base:namespace> <base:versionId> {fme:get-attribute("beginLifespanVersion")} </base:versionId> </base:Identifier> </cp:inspireId> <base:Identifier> <base:localId>595944</base:localId> <base:namespace>dk.kms.tn.roadnode</base:namespace> <base:versionId>2012-03-13T18:09:26</base:versionId> </base:Identifier> Example - geometry Coordinate System: ` EPSG:25832' Geometry Type: IFMEPoint Number of Geometry Traits: 1 GeometryTrait(string): `gml_id' has value `dk.kms.tn.roadnode.594897.20120803145439' Coordinate Dimension: 3 (725261.96,6187842.58,2.5) @GMLGeometry(TO_ATTRIBUTE, GML_3.2.1, gml_geom) <net:geometry> {fme:get-xml-attribute("gml_geom")} </net:geometry> <net:geometry> <gml:Point gml:id="dk.kms.tn.roadnode.594897.20120803145439” srsName="EPSG:25832" srsDimension="3"> <gml:pos>725261.96 6187842.58 2.5</gml:pos> </gml:Point> </net:geometry> gml:id • • • • • gmi:id is mandatory Unique within the document Must start with a letter In FME default is an UUID Build the same way as inspireId – <namespace>.<id>.<timestamp> gml:id NAVNE - ID - FraDato - Navn X MONTAGE - ID - FraDato - Geometri <gn:geometry> <gml:MultiCurve gml:id="dk.kms.gn.114294-0" srsName="EPSG:25832" srsDimension="2"> <gml:curveMember> <gml:LineString gml:id="dk.kms.gn.geom.24795424.20080613T085805"> <gml:posList>723071.8 6194337.45 .... </gml:LineString> </gml:curveMember> <gml:curveMember> <gml:LineString gml:id="dk.kms.gn.geom.24793173.20080613T085805"> <gml:posList>723273.34 6194566.81 ..... </gml:LineString> </gml:curveMember> ....... </gn:geometry> FME script design Design When we are designing an FME script we should reflect on • Schema mapping – FME or database • Generic or specific • Design of dataflow – design patterns • Pre- and post processing • Existing system architecture Dataflow design Dataflow design Dataflow design Conclusion design • Testing shows that design #3 is best – Design #1 will not work with big datasets because of the list – It use 10 to 15 percent less memory than #2 – The timing is identical to #2 – We can validate individual features Conclusion • After the first tests it been easy to work with templates • Schema mapping in Oracle with sql • A generic solution based on Design #3 • Templates as individual files read into an attribute • Only problem is FME can’t handle big datasets (+60.000 features) –yet Questions? Peter Laulund Rentemestervej 8 DK-2400 Copenhagen NV Denmark Phone: +45 72 54 51 73 E-mail: [email protected]