Data to Datafordeleren

Transcription

1 Data to Datafordeleren Infrastructure, architecture and design FME World tour Copenhagen 21 May 2015 SIDE 1

2 Who am I Peter Laulund Certified FME professional since 2007 Works at Geodatastyrelsen - The Danish Geodata Agency Working with FME since 1997 and with SpatialDirect / FMEServer since 2000 Recent Projects GST Data distribution system GST FMEServer infrastructure Data to Datafordeleren FME World tour Copenhagen 21 May 2015 SIDE 2

3 About Datafordeleren Common public solution for data distribution KMD won the contract GST contributes with Kortforsyningen It will replace Kortforsyningen by the end of 2017 GST data in Datafordeleren are, all our vector data, orthophoto, DHM, Lidar and more GST has to deliver delta data to Datafordeleren whenever our data are updated FME World tour Copenhagen 21 May 2015 SIDE 3

4 System architecture FME World tour Copenhagen 21 May 2015 SIDE 4

5 FME and Infrastructure Normally we see FME as a data transformer and translator, but FME also has a number of infrastructure tolls, that let us control the work and data flow before, during and after the translation A few examples are: Scripting with Tcl and Python HtppCaller and FTPCaller (new in 2015) WorkspaceRunner, FMEServerJobSubmitter, SystemCaller Reading and writing zip files With FMEServer we can do even more FME World tour Copenhagen 21 May 2015 SIDE 5

6 The Job Exchange format is XML Geometry as OGC well known text On the fly updating Data package contain updates and inserts Housekeeping Communicate with SOAP Move data to ftp FME World tour Copenhagen 21 May 2015 SIDE 6

7 XML example FME World tour Copenhagen 21 May 2015 SIDE 7

8 Workspace - 2xml FME World tour Copenhagen 21 May 2015 SIDE 8

9 Reading FME workspace select * from FME_READ.<tab> where gst_reg_id = <currentid> FME World tour Copenhagen 21 May 2015 SIDE 9

10 Reading database views FME World tour Copenhagen 21 May 2015 SIDE 10

11 Workspace main FME World tour Copenhagen 21 May 2015 SIDE 11

12 Workspace main summary Lookup sequence number and other metadata Extract data via a slave workspace Calculate md5 checksum Zip data and checksum Move zip file to an ftp server Mail error messages to administrator Log job status to the log database FME World tour Copenhagen 21 May 2015 SIDE 12

13 SystemCaller SystemCaller let you call a system command, a program or a bat file In this project we use it to calculate md5 checksum, zip data and move the zip file to the ftp server The setup is All commands are in attributes We have a bat file for each process We log the result as 1 (ok) and 0 (error) to the log database FME World tour Copenhagen 21 May 2015 SIDE 13

14 Workspace main, md5 custom transformer FME World tour Copenhagen 21 May 2015 SIDE 14

15 Error mail FME World tour Copenhagen 21 May 2015 SIDE 15

16 Workspace Batch FME World tour Copenhagen 21 May 2015 SIDE 16

17 Database trigger FME World tour Copenhagen 21 May 2015 SIDE 17

18 Log database FME World tour Copenhagen 21 May 2015 SIDE 18

19 Workspace design principles Generic workspace Separate logic data and metadata Place metadata, initiation, script etc. outside the workspace Control data flow with transformation attributes Let the database do the job Separate job into two or more workspaces master / slave arkitekture All jobs is logged to a database FME World tour Copenhagen 21 May 2015 SIDE 19

20 Conclusion The motto for the User conference in Vancouver was CONNECT, TRANSFORM and AUTOMATE, in this presentation I have illustrated how we can use FME and FMEServer for the entire workflow not only the data transformation FME is more than a Translator, it also have a number of infrastructure tools It is important how we design our projects best praxis FME and FMEServer will have critical role in moving data from GST to Datafordeleren FME World tour Copenhagen 21 May 2015 SIDE 20

21 Questions? For more information: Peter Laulund, Geodatastyrelsen - The Danish Geodata Agency FME World tour Copenhagen 21 May 2015 SIDE 21