ETL: CamptoCamp, FDO and Open Source - What about OLAP?

CamptoCamp & Talend


CampToCamp is introducing (not yet released, but anticipated), a Spatial ETL tool, that works in conjunction with Talend's Open Source ETL product Open Studio.


Once released, I'll begin to work with the product, but CamptoCamp was in Victoria presenting their solution at FOSS4G2007.


More information on their presentation can be found here.


Open Source ETL - Without Spatial


For Open Source ETL, without the Spatial, there are various Open Source solutions. Talend, being one of them.


You can take a look at:


1) Pentaho - URL: http://www.pentaho.com/

2) Clover - URL: http://www.cloveretl.org/

3) KETL - URL: http://www.ketl.org/


Is there another Open Source Spatial ETL Tool?

But we do have another option as well for the Spatial side now that AutoDesk is now working in the Open Source community and they released FDO (Feature Data Objects), which is similiar to FME - but is not an FME.


FDO is a Data Access Technology that was developed to manipulate, define and analyze geospatial data regardless of where it was stored.

FDO was originally developed and included in the Autodesk Map 3D 2005 product during the spring of 2004. In this initial implementation, it was capable of working with the following geospatial types:

• Oracle
• SDF

The following version introduced ArcSDE.

The third verision implemented more sources, and added providers for MySQL, SQL Server, ODBC, SHP, Raster, OGC WFS, and OGC WMS.

It was then that they decide to take FDO Open Source, but would not release the Oracle version - but being Open Source, there is always an option out there and some ingenious minds to come out with some solutions.

Quoting the OSGeo FDO History site:

"The release of FDO as open source coincided with the release of MapGuide as open source in 2006. It included the SDF, SHP, MySQL, ArcSDE, ODBC, OGC WFS, and OGC WMS providers. "

FDO was now out to the Open Source World - right on schedule with their release of MapGuide Open Source.

But what about Oracle and FDO?


Much of the work for the Oracle side has been developed by SL-King in Slovenia.

King.Oracle is Open Source FDO provider for Oracle.

Through this product, which is Open Source, SL-King is providing a tool that supports Oracle Locator and Oracle Spatial. It is specifically designed for Oracle and Oracle alone and they are designing it in such a way, that it will support full Oracle Spatial functionality.


Currently the latest version (Version 0.7.3) provides:

• Support for Oracle 10G, Oracle XE and Oracle 9i
• Optimized for Oracle
• Using plain Oracle tables and views
• Can be used inside AutoCAD MAP 3D to edit and query Oracle data

For a Flash Movie on this, please look here.

Can we convert between FDO different sources?

Yes, of course! This is Open Source.

Coming from SL-King, again, we have another ingenious tool, called FDO2FDO.

FDO2FDO is an Open Source FDO client application which uses the above mentioned Open Source FDO library to manipulate, create, and define geospatial data.

Currently, the software is capable of the following:

• Copy data from SHP files to SDF
• From SHP to Oracle
• Oracle to SDF...

In the end, FDO2FDO allows the user copy and modify any data from any FDO Data Store to any FDO Data Store.

There are three main parts in FDO2FDO and they are:

1. Fdo2Fdo Api library
2. F2Fcmd Command line utility
3. Fdo2Fdo GUI

An introduction to FDO2FDO can be found here.

There are always solutions out there.


Can Geospatial move towards ETL and Data Warehousing?


Yes.

Currently, in order to do web-mapping, you require the following:

• database
• web server
• data

That is all. You can have a web-map up-and-running with MapServer or MapGuide quite easily, definitely within less than a day depending on how complicated and stylish you want to get.

But look deeper and what is it we are after? The data.

Where is the data stored? In a database.

So why don't we work on bringing OLAP into the Internet mapping and GIS worlds?

Aggregations of data can occur anywhere.

You can group data by postal/post/zip codes, populations, etc. - this is prime data for OLAP.

Take industries such as oil and gas - well production is recorded on an hourly basis. This can be summed and aggregated into data marts (OLAP) for daily, monthly, yearly.

The maps are a starting point, but there should be no disconnect between the data, databases, GIS, internet mapping, as we are only working with data and transforming it into much more usable and valuable information.

By bringing OLAP into GIS and Internet mapping, you can add more value to your client's data and this data can be fed into other applications for reporting, etc., etc..

The internet map acts as a portal to a whole other world of information.

Just a few thoughts, as I'm involved in both worlds presently.

What do you think?

Feel free to write and let me know.

MapServer 5.0 Released - Enhancements & Performance Improvements

When I first started using Open Source for a project in Switzerland, I went immediately to the UMN MapServer. My first introduction to this was loading and using Shape Files to create a relatively simple internet mapping site - my first!





My initial impression of Open Source, was WOW! The ease, the speed and the amount of documentation and resources available made me a believer in the Geospatial Open Source world. I then moved onto loading the data into a PostGIS database developed by Paul Ramsey at Refractions Research here in Victoria. Again, WOW! This was not difficult. I had maps and a database up and running in less than a day and I was able to spatial queries as well.



Today, the Open Source Geospatial Foundation (OSGeo) proudly announced on their web-site a major release for UMN MapServer, Version 5.0.

In the Press Release it states:

"MapServer 5.0 is the first major release release since version 4.0 in July of 2003. While there have been regular releases every 6 months or so this is the first time developers felt the new feature set warranted the "major release" label."

Some of the new features include many small bug fixes, more enhancements and performance improvements.

A detailed listing of the new features include:

• style and label attribute attribute binding;
• lookup table-based raster color correction;
• dynamic charting (pie and bar); explicit label prioritizing;
• enhanced debugging and logging;
• dynamic allocation for layers, classes, styles and symbols;
• improved memory management and garbage collection for MapScript;
• numerous improvements for OGC service support.

This release also incorporates map rendering using the Anti-grain Geometry (AGG) graphics library. According to the Press Release this bring higher and better maps to the internet because of improved cartographic quality.

As the Press Release, and I concur, I to am "excited about the future possibilities of bringing high-end cartography to on-demand web mapping".

My hats off to Steve Lime and the UMN Developers worldwide!

MapGuide Open Source 1.2 Released & DM Solutions Fusion

MapGuide Open Source 1.2.0 has been released.

With this release of MapGuide Open Source, there have been enhancements that include:

1) Support for Unmanaged Data Sources
2) Cartographic Enhancements - Phase I

This includes the ability to use new symbols for Points and Labeling Linear Features (e.g. Highway Shields)


3) Feature Join Enhancements
4) Support coordinate system overrides on feature sources

More details can be found here.

Point 4 is interesting, as in MapGuide Open Source, there is within a feature source, an optional tag called SupplementalSpatialContextInfo. If an entry exists for a given Spatial Context, then the specified coordinate system override will be used. The coordinate system override will be given preference over the coordinate system defined in the data.

In the previous version, the SupplementalSpatialContextInfo was used only for data containing spatial contexts with undefined coordinate systems (I'll talk more about co-ordinate systems and map projections in future blogs). This SupplementalSpatialContextInfo would allow a coordinate system to be specified for a given spatial context. Essentially, the new SupplementalSpatialContextInfo can now be used to override coordinate systems for all spatial contexts in a feature source -- regardless of whether it contains a coordinate system or not.

Why did this modification come about?

Sometimes a feature source geometry may contain an incorrect coordinate system or does not have a coordinate system defined. What can be now? The above solution provides a convenient way to change the coordinate system without having to change the data of the feature source.
MapGuide Open Source handles spatial data with a specified coordinate system and with this modification it allows for the specification of a coordinate system override thereby letting the developer use data that in the past you may not have been able to use.

Symbols and Labels

Many GIS and Mapping applications are in need of very sophisticated symbolization. In the oil and gas industry, we see so many symbols and sometimes one symbol may have more than one definition (dry well, dry & abandoned, etc.).

In many parts of the world, countries published maps must adhere to high standards that are often even written into law. Amazingly though, sometimes even these maps are missing some of most standard information, such as Datum and Map Projection. The Cartographic Enhancements - Phase I introduces a high quality symbolization engine for MapGuide Open Source and allows for point symbols to be used as labels. This is especially pratical in the E&P industry.

Leaders Collaborating - MapGuide Open Source Fusion

As it can be seen Autodesk and the Open Source community are working together successfully in producing a user-friendly, practical, and powerful web-mapping tool.

DM Solutions and Autodesk have been working together on MapGuide Open Source Fusion. Dave McIlhagga and the DM Solutions group provide a very good description of Fusion here.

Take a look at how collaboration between these two companies have helped to move MapGuide ahead. Fusion is also available for UMN MapServer.

Open Source - A new software development paradigm?

Open Source development has been around for awhile, but it looks like it is gaining more momentum and with Autodesk's participation, we are seeing leaders in CAD and GIS realizing the advantages of working collaboratively with others to develop products that can be used and accepted by others.

I'll be talking more about Open Source Software development and some of the business models used and the many advantages of using Open Source in many different industries.