ETL: CamptoCamp, FDO and Open Source - What about OLAP?

CamptoCamp & Talend


CampToCamp is introducing (not yet released, but anticipated), a Spatial ETL tool, that works in conjunction with Talend's Open Source ETL product Open Studio.


Once released, I'll begin to work with the product, but CamptoCamp was in Victoria presenting their solution at FOSS4G2007.


More information on their presentation can be found here.


Open Source ETL - Without Spatial


For Open Source ETL, without the Spatial, there are various Open Source solutions. Talend, being one of them.


You can take a look at:


1) Pentaho - URL: http://www.pentaho.com/

2) Clover - URL: http://www.cloveretl.org/

3) KETL - URL: http://www.ketl.org/


Is there another Open Source Spatial ETL Tool?

But we do have another option as well for the Spatial side now that AutoDesk is now working in the Open Source community and they released FDO (Feature Data Objects), which is similiar to FME - but is not an FME.


FDO is a Data Access Technology that was developed to manipulate, define and analyze geospatial data regardless of where it was stored.

FDO was originally developed and included in the Autodesk Map 3D 2005 product during the spring of 2004. In this initial implementation, it was capable of working with the following geospatial types:

• Oracle
• SDF

The following version introduced ArcSDE.

The third verision implemented more sources, and added providers for MySQL, SQL Server, ODBC, SHP, Raster, OGC WFS, and OGC WMS.

It was then that they decide to take FDO Open Source, but would not release the Oracle version - but being Open Source, there is always an option out there and some ingenious minds to come out with some solutions.

Quoting the OSGeo FDO History site:

"The release of FDO as open source coincided with the release of MapGuide as open source in 2006. It included the SDF, SHP, MySQL, ArcSDE, ODBC, OGC WFS, and OGC WMS providers. "

FDO was now out to the Open Source World - right on schedule with their release of MapGuide Open Source.

But what about Oracle and FDO?


Much of the work for the Oracle side has been developed by SL-King in Slovenia.

King.Oracle is Open Source FDO provider for Oracle.

Through this product, which is Open Source, SL-King is providing a tool that supports Oracle Locator and Oracle Spatial. It is specifically designed for Oracle and Oracle alone and they are designing it in such a way, that it will support full Oracle Spatial functionality.


Currently the latest version (Version 0.7.3) provides:

• Support for Oracle 10G, Oracle XE and Oracle 9i
• Optimized for Oracle
• Using plain Oracle tables and views
• Can be used inside AutoCAD MAP 3D to edit and query Oracle data

For a Flash Movie on this, please look here.

Can we convert between FDO different sources?

Yes, of course! This is Open Source.

Coming from SL-King, again, we have another ingenious tool, called FDO2FDO.

FDO2FDO is an Open Source FDO client application which uses the above mentioned Open Source FDO library to manipulate, create, and define geospatial data.

Currently, the software is capable of the following:

• Copy data from SHP files to SDF
• From SHP to Oracle
• Oracle to SDF...

In the end, FDO2FDO allows the user copy and modify any data from any FDO Data Store to any FDO Data Store.

There are three main parts in FDO2FDO and they are:

1. Fdo2Fdo Api library
2. F2Fcmd Command line utility
3. Fdo2Fdo GUI

An introduction to FDO2FDO can be found here.

There are always solutions out there.


Can Geospatial move towards ETL and Data Warehousing?


Yes.

Currently, in order to do web-mapping, you require the following:

• database
• web server
• data

That is all. You can have a web-map up-and-running with MapServer or MapGuide quite easily, definitely within less than a day depending on how complicated and stylish you want to get.

But look deeper and what is it we are after? The data.

Where is the data stored? In a database.

So why don't we work on bringing OLAP into the Internet mapping and GIS worlds?

Aggregations of data can occur anywhere.

You can group data by postal/post/zip codes, populations, etc. - this is prime data for OLAP.

Take industries such as oil and gas - well production is recorded on an hourly basis. This can be summed and aggregated into data marts (OLAP) for daily, monthly, yearly.

The maps are a starting point, but there should be no disconnect between the data, databases, GIS, internet mapping, as we are only working with data and transforming it into much more usable and valuable information.

By bringing OLAP into GIS and Internet mapping, you can add more value to your client's data and this data can be fed into other applications for reporting, etc., etc..

The internet map acts as a portal to a whole other world of information.

Just a few thoughts, as I'm involved in both worlds presently.

What do you think?

Feel free to write and let me know.

Axis Order Confusion: Software, Geodesy & Transformations

Axis Order Confusion



Not simply X, Y, Z or N, E, S, W, but how we interpret them and use them in software, geodesy and navigation - it varies and has led to confusion among many people. This is an overview of the systems, the problem and points out things to take note of.



Over time, since the early days of mathematics, and then moving into software development, GIS and internet mapping, axis play an important role, whether it be in datum transformation or even displaying a map.


Is it X,Y,Z or Z,Y,X?


As we know in mathematics, a coordinate system is a system for assigning n-tuple of numbers or scalars to each point in n-dimensional space.



We are familiar with the following coordinate sytems involved in GIS/Mapping and Geodesy:

• Cartesian coordinate system, which may be called "rectangular", where for 3D space, it uses three numbers representing some distance


• Polar coordinate systems


• Curvilinear coordinates, which are based on an intersection of curves

Delving further into Polar coordinate systems, we see the following subtypes:

• Circular coordinate systems, which is represented by a point in a plane, by an angle, and a distance from the origin


• Cylinderical coordinate systems which require a point in space, a distance from an origin an a height


• Spherical coordinate system, which is represented by two angles and a distance from an origin

In mapping and geodesy, we deal quite often with Spherical coordinate systems and often refer to them as Geographic Coordinate Systems.

Ordered Pairs & Coordinate Systems

In GIS software and mapping software, we have three different perspectives for an Ordered Pair (2-tuple). They are in computer science, mathematics and of course Geographical Coordinate Systems.

Let's take a quick look at how these distinct perspectives see the their world:

In computer science and computer graphics, the axis order is (X,Y), where unsigned values increase to the bottom and to the right.

Mathematics sees this world differently for the same axis order (X,Y), where we have signed values increase to the right and upwards.

In the world, where we are most involved, Geographical Coordinate Systems, the axis order varies, sometimes being (X,Y) or (Y,X). The signed values increase upwards and to the right, based on a spheroid, hence we have -180, -90, 180, and 90.

Rotation Confusion as Well - Which Sign is positive?

There are 2 different conventions in use in the survey and mapping industry for defining rotations.

This too has led to considerable confusion in the GIS and mapping world.

Both are valid when used properly.

The two conventions can be referred to as:

1) Position Vector rotation (Commonly referred to as the Busra-Wolfe)

2) Coordinate Frame rotation

This essentially comes down to the left-handed vs. right-handed rotations (see image above) for the various transforms.

But what does this mean with left vs. right? Well, this is one way of determining orientation of axes and direction of rotations.

• Thumb = Positive X
• Index up = Positive Y
• Middle out = Positive Z

Clifford's Point of View

Clifford Mugnier of LSU, whom I met when I lived in Houston, gives a good explanation and way of handling rotations and coordinate systems. His reply can be found here.

My quote follows:

"Probably the best way to document a rotation method is:

use the accepted terms "coordinate frame" and "position vector".

These terms are also used in other disciplines like kinematics (robotics).

But there are more difficulties.One datum transformation method is laid down in the ISO 19111 standard.

This is an approximated 7-parameter Helmert transformation with position vectorrotation. See also ISO/IEC 18026 - Annex B.

PROJ uses about the same method, only the scaling method differs. PROJ does a scalar multiplication, ISO a matrix multiplication.

With the commonly used parameter ranges, the differences between the scaling methods are less than microns, so not important.

As far as I know, the Bursa-Wolfe transform is an approximation to theHelmert transform. The Helmert transform has sines and cosines in therotation matrices, whereas Bursa-Wolfe (and ISO 19111) use the angles themselves (since sin(a) ~ a, sin(a)*sin(b) ~ 0, and cos(a) ~ 1 for small angles).

If you read section B.6 of ISO/IEC 18026, then you'll notice that aBursa-Wolfe transform can be done with a position vector rotation model OR with a coordinate frame rotation model.

Just what one likes the best; be sure to use the correct sign of the rotation angles.

Therefore:

Bursa-Wolfe is NOT equivalent with this or that rotation model.

A well known expert repeatedly states that the Australians use the same datum transform rotation model as the Americans.

This is NOT true!

The order of the rotations differs (XYZ vs. ZYX).

See the Australian GDATechnical Manual.

By the way, if the rotations are approximated, then the order is not important.

Again, the differences in rotation order for real life numbers are literally microscopic."

He clearly points out, and I agree, that it is "silly to refer to a datum transformation method as an American, Australian, European, whatever regional model. If you want to document the way for instance an application transforms,givethe complete formulae, not just an ill-defined name. Why not referring to the EPSG coordinate transformation method numbers? These clearly define the most used datum transformation methods and projection methods."

A simple solution to a complex problem of rotations.

The key is to document and document and be sure to specify what is being done, what axis are being used and let your users know. Do not assume, ask questions, and your life will be easier when dealing with coordinate sytems and rotations.

Or even simplier, as Clifford points out, refer to the coordinate transformation method numbers referenced in the EPSG data and data model for defined coordinate sytems.

GeoTunis 2007 - November 15-17, 2007 - Tunis Science City

GeoTunis is occurring between 15th and 17th, November 2007 at the Tunis Science City in Tunisia.


As the website states: 'The task being an equal knowledge development and a stronger control of the digital information and telecommunications technologies with the purpose to decrease the digital gap between peoples. This symposium makes real the resolutions taken during the first national conference on map production "Geotunis 2006" and takes place in the same time with the International group world day celebration on the geographic information systems.'

OSGeo is hoping to be there as well, and information on OSGeo's participation can be found at here. This group is looking at promoting OSGeo and the Open Source Philosophy as it applies to Geospatial. They are also hoping to establish a stronger Francophone/French Speaking chapter that will include many French speaking nations.

I worked in Tunisia several years back with Schlumberger, involved with the Finder Data Management software and Tunisia's State Owned Oil Company - ETAP.

It is a beautiful country with great people and great food. The "Thé à la Menthe" and the "Chorba" are incredible.

With any luck, I'll make it to GeoTunis and be able to meet some new and old friends!

MapGuide Open Source 1.2 Released & DM Solutions Fusion

MapGuide Open Source 1.2.0 has been released.

With this release of MapGuide Open Source, there have been enhancements that include:

1) Support for Unmanaged Data Sources
2) Cartographic Enhancements - Phase I

This includes the ability to use new symbols for Points and Labeling Linear Features (e.g. Highway Shields)


3) Feature Join Enhancements
4) Support coordinate system overrides on feature sources

More details can be found here.

Point 4 is interesting, as in MapGuide Open Source, there is within a feature source, an optional tag called SupplementalSpatialContextInfo. If an entry exists for a given Spatial Context, then the specified coordinate system override will be used. The coordinate system override will be given preference over the coordinate system defined in the data.

In the previous version, the SupplementalSpatialContextInfo was used only for data containing spatial contexts with undefined coordinate systems (I'll talk more about co-ordinate systems and map projections in future blogs). This SupplementalSpatialContextInfo would allow a coordinate system to be specified for a given spatial context. Essentially, the new SupplementalSpatialContextInfo can now be used to override coordinate systems for all spatial contexts in a feature source -- regardless of whether it contains a coordinate system or not.

Why did this modification come about?

Sometimes a feature source geometry may contain an incorrect coordinate system or does not have a coordinate system defined. What can be now? The above solution provides a convenient way to change the coordinate system without having to change the data of the feature source.
MapGuide Open Source handles spatial data with a specified coordinate system and with this modification it allows for the specification of a coordinate system override thereby letting the developer use data that in the past you may not have been able to use.

Symbols and Labels

Many GIS and Mapping applications are in need of very sophisticated symbolization. In the oil and gas industry, we see so many symbols and sometimes one symbol may have more than one definition (dry well, dry & abandoned, etc.).

In many parts of the world, countries published maps must adhere to high standards that are often even written into law. Amazingly though, sometimes even these maps are missing some of most standard information, such as Datum and Map Projection. The Cartographic Enhancements - Phase I introduces a high quality symbolization engine for MapGuide Open Source and allows for point symbols to be used as labels. This is especially pratical in the E&P industry.

Leaders Collaborating - MapGuide Open Source Fusion

As it can be seen Autodesk and the Open Source community are working together successfully in producing a user-friendly, practical, and powerful web-mapping tool.

DM Solutions and Autodesk have been working together on MapGuide Open Source Fusion. Dave McIlhagga and the DM Solutions group provide a very good description of Fusion here.

Take a look at how collaboration between these two companies have helped to move MapGuide ahead. Fusion is also available for UMN MapServer.

Open Source - A new software development paradigm?

Open Source development has been around for awhile, but it looks like it is gaining more momentum and with Autodesk's participation, we are seeing leaders in CAD and GIS realizing the advantages of working collaboratively with others to develop products that can be used and accepted by others.

I'll be talking more about Open Source Software development and some of the business models used and the many advantages of using Open Source in many different industries.