Data Warehousing, Corporate Portal & e-Business Intelligence Applications
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
Mimno, Myers & Holum
"How to Get a Fast Payoff for Your Data Warehousing Investment"
FlashPoint Column - August 8, 2001
"How to Get a Fast Payoff for Your Data Warehousing Investment"
The tech slow-down is real and is affecting large numbers of users and providers of technology solutions. You've seen the recent headlines: "Morgan Stanley reports that 50% of tech buyers are cutting or slowing tech spending", "Growth in technology budgets of corporations has halved to about 6% this year", "CIOs are spending money only on small projects with quick payoffs". The primary concern of CIOs is "Will I get benefits from this in 90 days?"
If you are trying to get backing for a data warehousing project, how do you sell it to the CIO, CFO, and corporate managers in this challenging environment? If you are a provider of data warehousing products and services, how do you survive in this era of shrinking technology budgets?
The answer is to think smaller. How can you build a data warehousing project in small incremental steps that deliver a steadily increasing ROI? How can you get a measurable ROI in the first 90 days of your project? How can you implement the project using a bottom-up methodology that meets the requirement of the CFO for a fast payoff on the data warehousing investment?
Let's say that you want to build an enterprise data warehouse that will ultimately support 10 to 12 data marts, a central data warehouse, and, perhaps, an ODS. Your first issue is whether to use a top-down or bottom-up development methodology. The traditional, top-down approach typically requires a substantial, long-term effort to interview potential users of the data marts, document user requirements, and prepare a detailed enterprise data model for the data warehouse. This often involves in-depth business discovery across multiple business units, reconciliation of numerous differences in entity definitions and business rules, and months of work to specify an enterprise data model for the data warehouse. The top-down approach requires a lengthy, expensive, up-front development effort that synthesizes the requirements of multiple business units in order to define a model for the central data warehouse. In the current business climate, the top-down approach is likely to fail because it requires a large, up-front development expense and defers ROI.
In my experience, a more successful strategy is to use a bottom-up methodology that builds the data warehouse incrementally, one business unit at a time. The bottom-up development methodology may be used to build a data mart for a specified business area, such as sales, marketing, finance, etc., within a 90-day timebox. The bottom-up approach uses Rapid Application Development (RAD) techniques, rather than top-down Information Engineering techniques. Although the development effort is focused on building a single data mart, the data mart is embedded within a long-term enterprise data warehousing architecture that is specified in an early phase of the development methodology.
The assumption is that the long-term DW architecture will be implemented incrementally, one business unit at a time. The development of more complex components of the architecture, such as a central data warehouse and an ODS, are deferred until later stages of the development effort. The incremental development effort is kept under control through use of logical data modeling techniques (E-R diagrams that gradually expand to an enterprise model), and integration of all components of the architecture with central metadata, generated and maintained by the ETL tool.
Components of the architecture include multiple data sources (legacy files, RDBMSs, flat files, spreadsheets, ERP, CRM, Web server log files, etc.), an ETL tool, a central metadata repository, a metadata exchange architecture, a data modeling tool, a target database, and a BI tool. The initial development effort is a 90-day project resulting in the delivery of a fully functional data mart for a specified business area. The 90-day development effort begins on the day that the ETL tool, target database, and BI tool are installed successfully.
The bottom-up methodology, which is derived from RAD techniques, typically incorporates the following 12 steps:
1. Identify business drivers, sponsorship, risks, and ROI. Conduct survey of high-level business managers to identify specific, painful business problems, e.g. the organization is losing its competitive advantage, customers are moving to competitors, management has little insight and control over costs, consistent information cannot be shared across business units, etc.
2. Survey user needs and identify desired functionality. Unlike the top-down approach, the survey of user needs is very short (one day per business unit or less) and leads to the specification of a top-level data model for each business unit that will utilize a data mart. The top-level data models are then synthesized to identify common data sources, facts, dimensions, transformations, etc.
3. Design long-term, enterprise data warehousing architecture on paper. Following the survey of user needs, a workshop is convened whose function is to define the long-term vision for the data warehousing application, i.e., what will the architecture of the DW look like 2 to 3 years in the future
4. Define functional requirements for the initial subject area. A second workshop is convened to bring together the business users and development team for the first data mart to be implemented. Deliverables of the workshop include a preliminary project plan for the development effort, functional requirements for the first data mart, budget, required skill sets, etc.
5. Research and select DW components and tools. Following research, a short list of potential tools is defined, including ETL tools, data modeling tools, and BI tools. Selection of the finalists in each category may require a Proof of Concept test. Following selection and installation of the tools, the 90-day timebox is entered (steps 6-9 below)
6. Design target data base. Modeling of the target data base for the initial data mart proceeds through three steps: design of an entity-relationship diagram, then a logical dimensional model, and finally a physical model of the database schema
7. Build data mapping, extraction, transformation, and data cleansing rules. The data mapping and transformation rules are defined first in natural language, and then implemented using only the transformation objects supplied with the ETL tool. The objective is to avoid coding any extraction, transformation, or load processes
8. Build aggregation, summarization, partition, and distribution functions. The ETL tool is used to compute aggregates in one pass of the source data, using incremental aggregation techniques
9. Complete development of the initial architected data mart, using an exact subset of the enterprise data warehousing architecture. The deliverable at the end of the 90-day timebox is a fully functional data mart for the initial business unit
10. Build additional architected data marts. Additional data marts are built by a primary development team using common templates and components, such as conformed dimensions, common transformation objects, data models, central metadata definitions, etc.
11. Expand to an enterprise architecture, including a central data warehouse and an optional Operational Data Store. Development of the central data warehouse and ODS are deferred until they are clearly required. A central data warehouse is often required when detailed, atomic data from multiple data marts must be accessed to generate cross-business reports
12. Maintain and administer data warehouse. A secondary team may be used to enhance and maintain completed data marts. The primary team transfers transformation templates, data models, conformed dimensions, metadata, etc. to the secondary team to simplify the enhancement and administration of completed data marts
The bottom-up approach has the advantage that it requires little up-front investment and builds the application incrementally, proving the success of each step before going on to the next step. The first deliverable of the bottom-up approach is a fully functional data mart for a specific business unit. Subsequent data marts are delivered every 90 days or less. Complex components, such as the central data warehouse or the ODS, are deferred until they are clearly required, the team has accumulated sufficient experience, and the project has demonstrated significant ROI. In the bottom-up approach, the central data warehouse and the ODS are not on the critical path and may be deferred to a later development phase. The use of logical data modeling and metadata integration techniques ensures that all components of the application remain integrated, without the requirement for a central data warehouse.
I have used this development methodology successfully for a number of clients and have a good deal of experience with it. However, successful implementation of the methodology depends on several critical success factors, including a dedicated implementation team, consulting help at the beginning of the project, backing of a business manager who is hungry for a solution to a painful business problem, E-R data modeling, and integration of all components of the architecture with central metadata.
In summary, to meet the real-world need to get a rapid return for your data warehousing investment, think small - think bottom-up development.
For further information about the issues discussed in this report, please contact Pieter Mimno, Independent Consultant, at pmimno@mimno.com, or visit his Web site at www.mimno.com. Mr. Mimno specializes in the selection of system components and support for all phases of development for data warehousing, corporate portals, and eBusiness-intelligence applications.
|
[ TOP ] |