Data Warehousing, Corporate Portal & e-Business Intelligence Applications
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
How to Select an Extraction/Transformation/Load Tool #2
Stovepipe Data Marts
Unfortunately, there is a major limitation in the architecture of many ETL tools that has caused the failure of numerous data warehouses. The problem is lack of integration between central metadata maintained in the central ETL repository, and local metadata maintained in individual data marts.
Local metadata, which defines the local meaning of business data for end users, is maintained by end-user BI tools. However, many vendors of end-user tools do not support a metadata exchange architecture with ETL tools that can be used to integrate central metadata with local metadata. Lack of integration with central metadata results in the development of independent data marts, often called “stovepipe” data marts, which store local, inconsistent, incompatible definitions of business entities, semantics, and business rules. Stovepipe data marts support the needs of individual business units, but cannot be integrated at the corporate level because they do not to share data definitions and business semantics across the enterprise.
IS departments have known for years that they should not build stovepipe applications. In spite of the well-known problems with stovepipe applications, most data warehouses built today incorporate stovepipe data marts.
In their rush to deploy individual data marts, many organizations do not implement the infrastructure required to support consolidation of data across data marts. The result is a profusion of stovepipe data marts that satisfy the requirements of individual business units, but cannot be used to support corporate requirements for an integrated view of data across data marts. Subsequent attempts to install the infrastructure required to integrate data marts with a central repository often fail. The failure is due to the complexity and risk associated with making major changes to an operational data warehousing facility that is growing rapidly and supports large numbers of business end users.
It is extremely difficult to avoid building stovepipe data marts. To avoid stovepipe data marts, it is important to buy only components that integrate, out-of-the-box, with a central metadata repository generated and maintained by an ETL tool.
Architected Data Marts
The solution to the stovepipe problem is to build architected data marts that utilize a metadata exchange architecture to ensure that the local metadata maintained by each data mart conforms with definitions maintained in a central metadata repository. To support an architected data mart, an ETL tool is used to generate and maintain a central metadata repository. The central metadata repository provides a “single version of the truth” that can be used to define enterprisewide source data definitions, data models for target databases, and transformation rules that convert source data into target data. The metadata exchange architecture is used to maintain consistency between the central metadata and the local metadata.
The central metadata repository maintained by the ETL tool is at the heart of the data warehouse. It is an essential component of the architecture used to integrate all components of the data warehouse.
In practice, it is hard to find combinations of “best-of-breed” ETL tools and end-user tools that integrate at the metadata level. Each vendor of ETL tools generates proprietary metadata that is incompatible with metadata generated by other ETL tools. The lack of metadata standards makes it difficult to select best-of-breed tools from multiple vendors that support metadata integration within a common architecture.
Standards organizations, such as the OLAP Council and the Meta Data Coalition, have tried, but failed, to obtain industry-wide agreement on the definition of a metadata standard. Recently, in another attempt to define a metadata standard, the Meta Data Coalition merged with the Object Management Group. It is possible, but not likely, that the metadata standards being developed jointly by the merged organizations will be adopted by the industry.
|
[ TOP ] |