Showing posts with label Data Profiling. Show all posts
Showing posts with label Data Profiling. Show all posts

Sunday, November 22, 2009

My First Big Data Integration project in the early 1990’s

I am not sure how many people read this blog? However I have been reminiscing on some of my favorite data integration Projects/products from my past.  (I will outline this project and the type of work involved later., it taught me a lot about data integration from multiple related sources into a single normalized master version of the data, and then using that normalized view to output data in any shape required!)

I worked on this product very early in my professional career; The product basically allowed you to take data from any mainstream project management system, store it in a common normalized format, (think of a high level logical model of project management entities and relationships plus logic for common project management tasks like rolling-up or down project metrics and WBS) and then pushing out data after some processing, integration and normalization to any project system format that we supported.

Just for the record some of the products we supported at the time Microsoft Project, Primavera and Microsoft Excel to name but a few.

A key selling point at the time was the ability to take changes to a project plan in any of the major project management tools on the market, normalize them into a common format, do some project related processing to the plans, and then push out a normalized view of the project data (a single master copy if will) to end user system formats. Then once changes were made once more they were updated in the normalized logical model and then publish back out to supported systems once again.

It was pretty impressive for a good 12+ years ago!

Thursday, November 19, 2009

Data Quality or Data Profiling?

Well I started work on some more concepts for the match-Merge API. I also did a little work and analysis on data cleansing and data quality techniques and tools…

It has been interesting to see what is out there, and well it’s making me thing a little. It seems that data cleansing has been well thought out in most MDM and EDM solutions on the market, but there seems to be some confusion between ‘Data Profiling and Data Quality’.

I can see an overlap conceptually, but find it interesting that the definitions of data quality by various vendors is actually more on data profiling.

Wikipedia definition of Data Quality: http://en.wikipedia.org/wiki/Data_quality

Wikipedia definition of Data Profiling: http://en.wikipedia.org/wiki/Data_profiling

As you can see they are quite different.