Knowledge Engineers Limited LLP    

by Rand Losey

Intelligent Enterprise Magazine, March 1999

If your goal is enterprise-wide data sharing, an Operational Data Store (ODS) is what you need for revamped decision support.  We’re approaching the 21st century, but many organizations are exactly where they were 20 years ago.  The technology infrastructure supporting them is full of complex interfaces creating impediments to rapid and lasting business change.  Companies have consciously ignored staff warnings regarding building and maintaining this technological Tower of Babel.  Building an ODS can give your staff an integrated view of enterprise data organization for tactical decision support.  But it's critical to understand the ODS’s business value and the factors that can make or break your data store.

An ODS is a database environment that gives the tactical user an integrated, subject-oriented view of application system data stores.  As you extract relevant legacy system data, it's transformed into quality business information.  You eradicate inconsistencies by applying standard definitions, a common structure, and a defined set of extraction, transformation, and cleansing rules to each of the data elements residing in the ODS.

The ODS is a new name for an idea business and information technology expert James Martin originated and called the “subject database.”  In Martin's vision, databases are related to organizational subjects rather than application systems.  There should, for example, be a product database rather than separate order entry, inventory, and quality control databases relating to the product.  Many application systems could then use the same subject database.  Designing a stable, well-documented, and non-redundant data structure provides simpler and more elegant data processing.

A typical organization has a complex web of interrelated databases that impede change.  The ODS enables change because it organizes data around the enterprise’s data subjects.  Such organization permits multiple application systems to use the shared data resource.

An ODS pulls together, validates, cleanses, and integrates data from disparate source application systems, providing the end user community with an integrated view of enterprise data.  As a result, local operational departments can access the data for tactical decision support, day-to-day operations, and general reporting.

Constructing Your ODS

You wouldn't start building a skyscraper without first planning the entire building.  The architect has to meet with the customer to determine the structure’s specifications, which are translated into a set of architectural blueprints providing steelworkers, carpenters, electricians, and plumbers with the customer's vision.  When you have the overall plan, separate teams can go to work with the knowledge that the building components will fit together.

Similarly, the IS world is full of inspired trades-people (programmers, systems analysts, database designers, and system designers) who crave the challenge and excitement of solving problems with their own unique approaches.  They usually do an excellent job.  However, the data and process objects they use overlap substantially.

How do you create order from IS development and deployment chaos?  You start with a well-conceived plan that details the tasks, milestones, resources, time, and personnel required to design an enterprise business model.  The next step is to create an enterprise data model and an enterprise process model.  Finally, you design the relationships among data and process objects to create the enterprise model.

To implement the architecture specified in the enterprise model, you create a plan that provides executives and middle managers with a concise understanding of the current IS state as compared to the enterprise model's vision.  The plan drives and reports ongoing progress.  In terms of an approach, increments are most effective for creating an ODS environment, ensuring that you’re building only currently required components.  You craft the logical ODS architecture, recognizing the enterprise-wide vision.

The first ODS implementation will create the first subject area data objects deployed as part of the ODS environment.  After initially deploying the environment, the team should focus on creating the next iteration.

Each successive ODS environment iteration should benefit from reusing knowledge gained in previous iterations, experience, and even currently deployed data objects.  You should deploy each successive iteration to end users on a quarterly basis.  In order to meet this schedule, the ODS team must ensure that the scope of the new data objects included in the environment is manageable within the quarter.

Data Boundaries

All matter is built on the foundation of a small number of atomic building blocks.  If you can label and construct matter from a few atomic constructs, why is it that data processors can't construct and label such data based on a finite number of “core” data elements?  Why is it that a typical IS shop’s source code libraries contain hundreds or even thousands of data element names?  Are all data molecules formed from a finite number of data atoms?

Fortunately, you can reduce these bloated amounts of data element names.  The majority of names are homonyms, synonyms, and aliases created in the absence of three factors: naming standards, an enterprise data architecture perspective, and planning for data creation and use.  Many programmers, analysts, and database specialists create data element names and label their own data without regard for the way others in the enterprise do so.

Using the ODS construct, your organization can design and implement a solid foundation of core data elements and entities from which you can derive reusable, accurate, integrated, and reliable information.

Executing Your ODS

It's critical that you pay attention to five primary project factors to ensure ODS success:

Direct participation by top management.  To implement an integrated, cross-functional ODS; it's imperative to gain corporate top management commitment.  In most organizations, data integration projects face a great deal of resistance in and outside the IS department.  Data processors are a creative lot by nature who enjoy the challenge of unraveling the nightmare world of data interfaces.  In addition, end users want exclusive rights over their own data.  Top management must motivate both groups to jump aboard the ODS project, and it must participate directly in the ODS project’s planning and quality view.  Without this commitment level, you might deal with political roadblocks.

Well-defined scope.  The ODS team must have the time and energy to learn about building an ODS without being encumbered by in-house politics.  In other words, the team must learn to walk before it runs.  First time, large-scale ODS projects have a tendency to fail due to political and organizational conflicts.

Design stability.  Stability doesn't imply that the ODS will never change; you can make most changes without requiring anyone to rewrite application systems.  The ODS's underlying logical data model should produce physical tables independent of their physical implementation on current hardware and systems software.  As the underlying technology changes over time, the ODS’s logical structures will remain valid.

Data object abstraction and generalization.  You must design the ODS’s logical data model with an appropriate data normalization level; as a result, you'll be able to add entities and attributes to the model without rewriting existing applications.  Third normal form is the recommended normalization level that goes hand-in-hand with the data abstraction and generalization principles you're applying.

A data modeling team with one or two members.  Integrating data across business areas represents the ODS’s primary value.  Many users think that the time it takes to integrate the data into a coherent data model results in additional delay for little return.  This perception is erroneous.  The efforts of one or two data modelers over the span of a few weeks results in a coherent model.  Enterprise level data modeling is not a multithreading discipline.

Sharing Data

Data administrators typically have difficulty convincing management that information is an asset of the entire organization - not the private property of individual operating units exerting tremendous political pressure on IS staff to build something at the expense of enterprise needs.  It's imperative that you draft an enterprise information architecture strategy that's tied directly to the business strategy.  You should also constantly keep an eye toward having enterprise data architecture and support for business units; otherwise, the ODS initiative will fail, and IS will end up with “silo” data stores that support the tactical needs of the business units rather than the enterprise strategy.  Following these guidelines will ensure your ODS evolves into a critical enterprise database.

When you've conquered the politics of enterprise data modeling, created a robust enterprise data model, and populated your ODS, the DBMS software provides the capabilities necessary to share data across the enterprise.  The DBMS not only ensures that there will be data integrity, security, concurrency, and high ODS availability, but it also provides fault-tolerant backup and recovery, performance monitoring, and system management capabilities.

Data management theorists and practitioners have aimed for enterprise-wide data sharing for more than 25 years.  With the ODS construct; we've taken bold steps in our pursuit of this goal.  The ODS should evolve into the physical instantiation of the enterprise data model.

Migrating to an ODS

You can't implement an ODS in a single release, nor will all existing legacy systems migrate to it at the same time.  You should implement the ODS in stages.  But you have to factor in a few issues into choosing the optimal migration strategy.  There is your return on investment, for one thing.  The migration strategy should follow a pay-as-you-go philosophy and provide tangible benefits to the organization as quickly as possible.  The strategy should minimize both business and technical risks.  Avoiding unproven new technology, for example, ensures that your basic architectural core and concepts will be sound.

Migration to enterprise-wide ODS capability from existing application-specific databases shouldn't be an end in itself.  Rather, migration should take place as part of two initiatives: increments that follow operational systems portfolio restructuring implementing new a cross-departmental or cross-functional application systems, and increments that respond to departmental and enterprise data requirements simultaneously.

You must carefully evaluate the degree of cultural change on the IS staff, the user community, and the organization's customer base.  You shouldn't let the rate of change be excessively rapid or drastic.  Implementing an ODS should be evolutionary - not revolutionary.

ODS Business Value

It's difficult to dispute the value of having subject-oriented data at the corporate level, giving you a common view and understanding of data regardless of the business function.  A customer is a customer, regardless of who's asking the question.  This shared data environment offers more accurate and complete information for improved decision-making.  The ODS offers several benefits that will add business value to your company.

Understanding the fundamental health of the enterprise: Many Fortune 1000 companies can’t determine how many customers they have.  They also can’t figure out whether they're making or losing money on any one product or service.  Normally, profit and loss metrics are accurate at only the profit center level.  Rolling up these figures to the corporate level is impossible for most organizations.  Enterprises that have built an ODS find themselves able to compete more successfully than their competitors.

Providing a reference data repository: One central component of an information architecture is reference data.  The ODS lets you create cross-functional subject areas (customer, product, and agreement, for example.)  One organization reported a significant reduction in accounts receivable merely because the correct billing addresses were on a reference database.  The ODS is the ideal place to create central reference data that are shared among application systems.

Speeding up application system development: As you add more enterprise subjects to the ODS, the application development rate increases because the data already exists.

Providing a data staging area: As the enterprise’s data warehouse architecture evolves, the ODS becomes paramount to integrating and staging data from disparate legacy systems.  The data is subsequently distributed to data marts or enterprise data warehouses and gives you uniform operational reports.

The ODS Pays for Itself

Our preoccupation with unraveling the decades of “silo” application system and data store design shouldn't surprise anyone.  Organizations have dug themselves into a deep “data hole” that can only be filled by designing and implementing an enterprise data architecture.  The ODS offers a significant return on investment primarily because it supports creating reusable data.  You build data once, and you can share it many times across many applications.

An ODS is integral to solving integrity problems many organizations face.  From an IS perspective, managing many systems and redundant data stores is a difficult and complex endeavor.  From a business perspective, identical queries often yield different answers, because each functional area has its own data.

Unfortunately, most people want immediate gratification.  It's worse in the business world, because you have to perform for the bottom line at the end of every quarter.  Increasing flexibility and reducing time to market won't happen by accident through technology acquisitions, software packages, or custom developed systems.  It will only happen if you invest in design and implementing an enterprise data architecture. 

A Win-Win Data Solution

Most late 20th-century organizations are finding that corporate survival requires shifting course.  By creating an ODS based on your organization’s fundamental data building blocks, you'll provide a foundation on which to respond rapidly to the changing demands placed on your business into the next millennium.

The shift from building departmental solutions to building enterprise solutions will require major cultural changes within the enterprise's business and IS departments.  To compete and win in the next century, you have to discontinue building business systems and data stores that reflect management organization.  Instead, focus on future business systems and data stores that reflect the true needs inherent in delivering products and services to the customer.

by Rand Losey and Michael Schroeck

Data Management Review Magazine, January 2001

As companies aggressively pursue their new economy business strategies, they are quickly realizing the importance of having ready access to information across their complete value chain. So whether it's implementing integrated electronic customer relationship management (e-CRM), business-to-business (B2B), employee portals or a balanced scorecard, the need for integrated, granular information is more important than ever. As a result, we are seeing a significant resurgence relative to the importance and value of the operational data store (ODS).

Most organizations today, both traditional brick and mortar and dot-coms, have a complex network of nonintegrated databases that impact their ability to rapidly respond to market changes. When information about a company's products, services, customers, employees, etc., is spread across various computer systems with different file formats, update frequencies and data access availability, it impedes the ability to respond to changing market conditions.

We have found that an ODS can be utilized as an important enabler of change within an organization, storing and organizing data around common subjects ­ organization, product, geographic location, assets, customers ­ across the enterprise.

Operational Data Stores Defined

The ODS usually evolves with the development of the enterprise data model.

An ODS is an environment that pulls together, validates, cleanses and integrates data from disparate source application systems. This becomes the foundation for providing the end-user community with an integrated view of enterprise data to enable users anywhere in the organization to access information for strategic and/or tactical decision support, day-to-day operations and management reporting.

As relevant legacy system data is extracted, it is transformed into quality business information. Inconsistencies are eradicated by applying standard definitions, a common structure and a defined set of extraction, transformation and cleansing rules to each of the data elements that reside in the ODS.

An ODS Versus a Data Warehouse

The ODS, while subject oriented and integrated like a data warehouse, is actually quite different. The following table establishes the major differences between these two technology solutions.

 

Figure 1: ODS/Data Warehouse Comparison

The ODS's Value Proposition

The benefits associated with an ODS span all industries. Organizations that have created an ODS report the following results:

• Enhanced ability to perform what-if analyses of project costs and revenues associated with new service and product launches.
• Improved accessibility to critical operational data.
• Shortened time required to build a data warehouse (integrated data already resides in the ODS).
• Integrated customer and product data in the ODS helps organizations evolve into a customer-centric focus from an account-centric focus.
• Ability to analyze product and service usage data on a near real-time basis.

The key drivers for using the ODS as an integral part of an organization's IT strategy include:

Developing an accurate understanding of the state of the enterprise. Many companies cannot determine how many customers they have, or their profits and losses on specific products or services at a corporate level. With an ODS, organizations can have a holistic view of their financial metrics and customer transactions, helping end users better understand the customer and make well-informed business decisions.

Improved performance associated with generating operational reports from the ODS as opposed to the transaction/legacy systems.

Providing a reference data repository. Reference data is an important part of the data architecture. The ODS enables the creation of a "single version of the truth" for key, cross- functional subject areas. One organization reported a significant reduction in accounts receivable simply by having correct billing addresses on a reference database. The ODS is the ideal place to create central versions of reference data that are then shared among different application systems.

Serving as the data staging area. As the enterprise's data warehouse architecture evolves, the ODS becomes key to the integration and staging of data from disparate legacy systems. The data is then distributed to data marts or an enterprise data warehouse.

Planning an ODS

A well- researched plan is essential to any successful information system implementation ­ one that is aligned with the organization's overall business strategy. Once a plan is in place, enterprise data and process models must be created, establishing the relationships between data and process objects to create the enterprise information architecture. This process will ensure that the ODS implementation is aligned with the overall business strategy.

Organizations must carefully manage the development and implementation of the enterprise data architecture to avoid having silo data stores that support the tactical needs of a single business unit rather than the strategic needs of the enterprise.

Guiding Principles of an ODS

An incremental approach to the development of the ODS has proven to be the most effective path to success. The incremental approach ensures that organizations contain the scope and build components to deliver short-term value consistent with the logical ODS architecture that is crafted based on the ultimate enterprise-wide vision of the organization. Additionally, the ODS should be implemented in stages relative to integrating the various source systems. As the ODS evolves, existing legacy systems should mi-grate to it in a planned fashion.

Before choosing a migration strategy, organizations must keep in mind two im- portant factors:

• Realizing return on investment. The strategy should provide tangible benefits to the organization as quickly as possible.
• Managing risks. The migration strategy should minimize both business and technical risks to the extent possible.

Achieving the Ultimate Goal

Sharing information across the enterprise has been the ultimate goal of data management theorists and practitioners for more than twenty-five years.

Most organizations are finding that corporate survival requires the flexibility to adapt to dynamic market demands. By creating an ODS as the centerpiece of your information architecture, you will provide a strong foundation upon which to rapidly respond to the changing demands placed on your business in the new millennium.

Rand Losey, a Director in Pricewaterhousecoopers’ Global Data Warehousing Practice, was a major contributor to this "Insights from the Front Line" monthly column.

Michael Schroeck, a Partner in Pricewaterhousecoopers’ Global Data Warehousing Practice, was a major contributor to this "Insights from the Front Line" monthly column.

by Rand Losey

Data Management Review Magazine, January 2003

A weak strategic plan for an enterprise data warehouse (EDW) environment can result in untold millions being wasted in rework. That does not even include lost opportunity to increase bottom-line dollars to an organization via incisive analytics and reporting. While frightening, the good news is that projects can be planned and implemented in a judicious manner that will deliver results with speed and certainty.  The typical EDW initiative is driven by the business need for organizations to compete at a global level. This requires data standardization, data conformance and data sharing.  An EDW sets the foundation for sharing data across the enterprise in a controlled and managed environment.

Corporate Strategies and Goals

Most Fortune 1000 organizations share the following key corporate strategies and goals:

• Producing the highest quality products,

• Focusing on high-growth, high-margin segments,

• Investing in high-potential markets and brands,

• Building organizational excellence and improving cost structure and efficiencies, and

• Pursuing strategic opportunities.

Your organization will effectively address these priorities through the development of a comprehensive and effective EDW environment.  An EDW will support the creation of top-notch analytics and business intelligence to answer the questions needed to meet these priorities. Implementing such a solution can take the form of a project-based, independent data mart approach or be done in a strategic manner by first developing an EDW strategy document.

A well thought-out EDW strategy helps your organization:

• Gain proper high-level business sponsorship.

• Provide the tools and processes required for IT to effectively manage the complexities of your organization’s most precious asset, its information.

• Set the foundation for sharing data across the enterprise in a controlled and managed environment.

• Provide a consistent enterprise-wide view to the end user ensuring data integrity and reliability across systems at a global level.

• Gain maximum advantage from its enterprise resource planning (ERP) and legacy systems investment through data integration at an enterprise level.

• Set the stage for comprehensive enterprise-wide business intelligence and the prodigious return on investment that such analytic capability can deliver to your organization in meeting its top management goals.

Not having an EDW strategy will result in a piecemeal approach to data warehousing efforts. This will lead to inconsistency, duplicity, disparate data warehouses that may give conflicting answers to basic questions (loss of data integrity), thereby negating the usefulness of a data warehouse and frittering away opportunities to maximize scarce IT dollars. Furthermore, it also results in loss of opportunity to add to your organization’s bottom-line dollars due to inability to make sound and correct business decisions. The consequences of a short-term, independent data mart approach strategy can be far-reaching to the organization. The negative effects of such a strategy have been well documented in industry journals and research studies by leading research organizations such as AMR, META Group, and Gartner, Inc.

Questions Addressed

An EDW strategy will guide the design and deployment of the solution.  It should answer the following questions:

• Why are you building an EDW environment?

• What is the scope of the EDW environment?

• Who are the stakeholders of the EDW environment?

• Who are the business sponsors of the EDW environment?

• What are the business objectives and goals that the EDW environment will help achieve?

• What business problems will and will not be addressed by the EDW environment?

• What are the high-level requirements for business use of enterprise data?

• What are the risks involved and how will they be managed?

• What is your organization’s enterprise application integration strategy? How does it affect and/or how can it be leveraged for the EDW environment?

• What is your organization’s information delivery strategy? Who are the end users both external (customers, suppliers, partners) and internal? How will data be delivered to these end users?

• How will the EDW environment interface with and potentially support the existing (or those being retired) portfolio of decision support system (DSS) applications?

• What is the migration strategy for moving existing data warehouses/marts to the EDW environment?

• What is the implementation road map or detailed project plan on how to arrive at the end-state vision of your EDW?

• Is your organization ready to build an EDW environment from both business and technology perspectives?

Strategy Definition Activities

In order to address these questions, the following activities need to occur. It is our experience that these activities take anywhere from four to six weeks. The duration is greatly influenced by the availability of the key business and IT leaders. As with any strategic activity, participation of senior executives is key. You must make a strong  business case to senior management to obtain the necessary level of executive commitment and participation.

An effective EDW strategy will include the following deliverables:

• Business Strategy to EDW Strategy Mapping:  This deliverable documents your organization’s business strategy including a statement of business direction, mission, goals and objectives. Mapping the EDW strategy to the business strategy will help ensure that the business vision of the executive management team is factored into the resultant DSS environment.

• Scope Statement:  This deliverable defines the boundaries of the proposed EDW environment.  It documents the business units and business areas that will benefit from the EDW, including external partners.  It documents the business functions that will be supported or the decision making that will be enabled by the EDW.  It also defines the business operational areas that will feed data to the EDW.

• EDW Stakeholders:  This deliverable determines key organizational stakeholders – those who have the most to win or lose by the implementation of the EDW.  It identifies why they are stakeholders and their expectations.

• Business Goals and Objectives:  This deliverable identifies the business reasons for building and maintaining the EDW.  These goals are constrained and measurable statements about the effort.  It documents the business rationale and business drivers that are driving the EDW initiative.

• Business Benefits:  This deliverable identifies the business benefits associated with deploying and maintaining an EDW.

• Critical Success Factors:  This deliverable lists specific overriding factors and accomplishments necessary within the EDW initiative that must be met to consider the effort a success.

• Risk Factors:  The failure to address the critical success factors gives rise to certain risks.  In addition, there are other risks that may impact the success of the EDW initiative and the realization of the goals and objectives of the project.  These risks are documented along with suggested mitigation.

• EDW Organization Statement:  This deliverable addresses the strong and natural tendency of isolated reporting environments, either by function or by geography.  The listener always acknowledges the value of information integration, as long as it is understood that it is the listener who will be doing the integration, not some central bureaucratic organization.  This deliverable will identify the most effective enterprise data warehousing organization.

• Existing Data Warehousing Strategy:   This deliverable provides a set of matrices mapping the currently designed and/or deployed data objects to the business intelligence processes supported.  For example, customer data may be stored in departmental data marts redundantly.  This information will provide an understanding of the extent of DSS “silos” built to date.

• Information Delivery Strategy:  This deliverable describes how your organization will share information collaboratively with customers, suppliers and other external partners. It also describes what kinds of information and the manner in which it will be made available to internal users of your organization.

• Enterprise Application Integration Strategy:  This deliverable defines the standards being adopted by your organization regarding the movement of transactional data between different OLTP applications and external partners and also within the EDW environment.

• Data Architecture Strategy:  This deliverable describes the approach to be followed when modeling the various levels of the EDW environment (conceptual, logical and physical.)

• Meta Data Management Strategy:  This deliverable describes the approach to be followed when facing the challenge of maintaining and synchronizing the various tools that create and maintain their own meta data. With each extraction, administration, and query tool creating and maintaining its own set of meta data, your organization faces tough challenges in delivering integrated and consistent enterprise level meta data to end users.

• Migration Strategy:  This deliverable defines a migration strategy for the existing DSS systems.  An EDW environment will not be implemented in a single release nor will all existing legacy DSS systems migrate to it at the same time.  The most prudent approach is to follow the iterative development paradigm.  It also discusses the retirement of existing DSS solutions which do not enable an enterprise view of information.

• EDW Implementation Road Map:  This deliverable produces a detailed project plan used to incrementally arrive at the end-state vision of your EDW.

Turning Vision into Reality

Successful EDW implementations are dependent on having firm senior management commitment and involvement to:

• Enable business process transformation.

• Enable organizational and cultural changes.

• Empower employees.

• Articulate key performance indicators (KPIs) by enterprise, business unit and subject area.

• Dedicate action-takers on both a tactical and strategic level to react quickly to KPI drift.

• Project funding, leadership and ongoing internal communication.

Properly done, with shared values and commitment, your organization will join those firms that have successfully reaped the benefits of an EDW.  Improperly done, extensive efforts will result in enormous loss of opportunity, tremendous cost, discouragement and frustration.

by Rand Losey

Information Management Magazine, January 2004

If your goal is enterprise-wide data sharing, an enterprise data architecture (EDA) is what you need for revamped decision support. As we enter the 21^st century, many organizations are exactly where they were 20 years ago. The technology infrastructure supporting them is full of complex interfaces creating impediments to rapid and lasting business change. Companies have consciously ignored staff warnings regarding building and maintaining this technological Tower of Babel. Building an EDA can give your staff an integrated view of enterprise data organization for strategic and tactical decision support. However, it's critical to understand the business value of an EDA and the factors that can make or break your EDA.

A typical organization has a complex web of interrelated business systems and databases that impede change. An EDA enables organizational change because it organizes data around the enterprise's data subjects. Such organization permits multiple application systems to use the shared data resource. An EDA pulls together, validates, cleanses and integrates data from disparate source application systems, providing the end-user community with an integrated view of enterprise data. As a result, operational departments can access the data for strategic and tactical decision support, day-to-day operations and general reporting.

Constructing Your Enterprise Data Architecture

You wouldn't start building a skyscraper without first planning the entire building. The architect must meet with the customer to determine the structure's specifications, which are translated into a set of architectural blueprints providing steelworkers, carpenters, electricians and plumbers with the customer's vision. With the overall plan, separate teams can go to work with the knowledge that the building components will fit together.

Similarly, the information technology (IT) world is full of inspired tradespeople (programmers, systems analysts, database designers and system designers) who crave the challenge and excitement of solving problems with their own unique approaches. They usually do an excellent job. However, the data and process objects they use overlap substantially.

How do you create order from IT development and deployment chaos? You start with a well-conceived plan that details the tasks, milestones, resources, time and personnel required to design an enterprise business model. The next step is to create an enterprise data model and an enterprise process model. Finally, you identify the relationships among data and process objects to create the enterprise business model. Figure 1 shows typical data subject areas.

 

 Figure 1: Enterprise Data Subject Areas

To implement the EDA specified in the enterprise business model, you create a plan that provides executives and middle managers with a concise understanding of the current IT state as compared to the enterprise model's vision. The plan drives and reports ongoing progress. In terms of an approach, increments are most effective for creating an EDA environment, ensuring that you're building only currently required components. The first EDA implementation will create the first subject area data objects deployed as part of the EDA environment. After initially deploying the environment, the team should focus on creating the next iteration.

Each successive EDA environment iteration should benefit from reusing knowledge gained in previous iterations, experience and even currently deployed data objects. You should deploy each successive iteration to end users on a quarterly basis. In order to meet this schedule, the EDA team must ensure that the scope of the new data objects included in the environment is manageable within the quarter.

Data Boundaries

All matter is built on the foundation of a small number of atomic building blocks. If you can label and construct matter from a few atomic constructs, why is it that data processors can't construct and label such data based on a finite number of "core" data elements? Why is it that a typical IT shop's source code libraries contain hundreds or even thousands of data element names? Are all data molecules formed from a finite number of data atoms?

Fortunately, you can reduce these bloated amounts of data element names. The majority of names are homonyms, synonyms and aliases created in the absence of three factors: naming standards, an EDA perspective and planning for data creation and use. Many programmers, analysts and database specialists create data element names and label their own data without regard for the way others in the enterprise do so.

Using the EDA construct, your organization can design and implement a solid foundation of core data elements and entities from which you can derive reusable, accurate, integrated and reliable information.

Executing Your Enterprise Data Architecture

It's critical that you pay attention to five primary project factors to ensure EDA success:

• Direct participation by top management. To implement an integrated, cross-functional EDA, it's imperative to gain corporate top management commitment. In most organizations, data integration projects face a great deal of resistance within and outside the IT department. Data processors are a creative lot by nature who enjoy the challenge of unraveling the nightmare world of data interfaces. In addition, end users want exclusive rights over their own data. Top management must motivate both groups to jump aboard the EDA initiative. Additionally, top management must participate directly in the initiative's planning and quality view. Without this commitment level, you might encounter political roadblocks.
• Well-defined scope. The EDA team must have the time and energy to learn about building an EDA without being encumbered by in-house politics. In other words, the team must learn to walk before it runs. First time, large-scale EDA projects have a tendency to fail due to political and organizational conflicts.
• Design stability. Stability doesn't imply that the EDA will never change; you can make most changes without requiring anyone to rewrite application systems. The EDA's underlying logical data model should produce physical tables independent of their physical implementation on current hardware and systems software. As the underlying technology changes over time, the EDA's logical data structures will remain valid.
• Data object abstraction and generalization. You must design the EDA's logical data model with an appropriate data normalization level; as a result, you'll be able to add entities and attributes to the model without rewriting existing applications.
• A data modeling team with one or two members. Integrating data across business areas represents the EDA's primary value. Many users think that the time it takes to integrate the data into a coherent data model results in additional delay for little return. This perception is erroneous. The efforts of one or two data modelers over the span of a few months will result in a coherent model. Enterprise-level data modeling is not a multithreading discipline.

Sharing Data

Data administrators typically have difficulty convincing management that information is an asset of the entire organization –­ not the private property of individual operating units exerting tremendous political pressure on the IT staff to build something at the expense of enterprise needs. It's imperative that you draft an enterprise data architecture strategy document that's tied directly to your organization's business strategy.

When you've conquered the politics of enterprise data modeling, created a robust enterprise data model and populated your enterprise-level databases, the DBMS (database management system) software provides the capabilities necessary to share data across the enterprise. The DBMS not only ensures that there will be data integrity, security, concurrency and high enterprise-level database availability, but it also provides fault-tolerant backup and recovery, performance monitoring and system management capabilities.

Data management theorists and practitioners have aimed for enterprise-wide data sharing for more than 25 years. With the enterprise-level database construct, we've taken bold steps in our pursuit of this goal. As shown in Figure 2, the enterprise-level database should evolve into the physical instantiation of the enterprise data model.

 

 Figure 2: Enterprise Data Architecture

Migrating to an Enterprise Data Architecture

You can't implement an EDA in a single release, nor will all existing legacy systems migrate to it at the same time. You should implement your EDA in stages. However, you must factor a few issues into choosing the optimal migration strategy. There is your return on investment, for one thing. The migration strategy should follow a pay-as-you-go philosophy and provide tangible benefits to the organization as quickly as possible. The strategy should minimize both business and technical risks. Avoiding unproven new technology, for example, ensures that your basic architectural core and concepts will be sound.

Migration to an EDA from existing application-specific databases shouldn't be an end in itself. Rather, migration should take place as part of two initiatives:

• Increments that follow operational systems portfolio restructuring, implementing new cross-departmental or cross-functional application systems.
• Increments that respond to departmental and enterprise data requirements simultaneously.

You must carefully evaluate the degree of cultural change on the IT staff, the user community and your organization's customer base. You shouldn't let the rate of change be excessively rapid or drastic. Implementing an EDA should be evolutionary –­ not revolutionary.

Enterprise Data Architecture Business Value

It's difficult to dispute the value of having subject-oriented data at the corporate level, giving you a common view and understanding of data regardless of the business function. A customer is a customer, regardless of who's asking the question. This shared data environment offers more accurate and complete information for improved decision making. An EDA offers several benefits that will add business value to your company. The architecture will:

• Provide an understanding of the fundamental health of the enterprise. Many Fortune 1000 companies can't accurately determine how many customers they have. They also can't figure out whether they're making or losing money on any one product or service. Normally, profit and loss metrics are accurate only at the profit center level. Rolling up these figures to the corporate level is impossible for most organizations. Enterprises that have built an EDA find themselves able to compete more successfully than their competitors.
• Speed application system development. As you add more enterprise data subjects to the EDA, the application development rate increases because the data already exists.
• Provide the tools and processes required for IT to effectively manage the complexities of your organization's most precious asset, its information.
• Set the foundation for sharing data across the enterprise in a controlled and managed environment.
• Provide a consistent enterprise-wide view to the end user ensuring data integrity and reliability across business systems at a global level.
• Increase the ability of your knowledge-workers to transform data into information.
• Provide a standard framework for addressing data issues and data design decisions in order to develop durable business solutions.
• Enable your organization to have a high degree of control over the replication and duplication of data.
• Enable your business units to process information in a more cost-effective fashion.

An Enterprise Data Architecture Pays for Itself

Our preoccupation with unraveling the decades of "silo" application system and data store design shouldn't surprise anyone. Organizations have dug themselves into a deep "data hole" that can only be filled by designing and implementing an EDA. An EDA offers a significant return on investment primarily because it supports creating reusable data. You build data once, and you can share it many times across many application systems.

An EDA is integral to solving the data integrity problems many organizations face. From an IT perspective, managing many application systems and redundant data stores is a difficult and complex endeavor. From a business perspective, identical queries often yield different answers because each functional area has its own data.

Unfortunately, most people want immediate gratification. It's worse in the business world because at the end of every quarter, you must perform for the bottom line. Increasing flexibility and reducing time to market won't happen accidentally through technology acquisitions, software packages or custom developed systems. It will only happen if you invest in the design and implementation of an EDA.

A Win/Win Data Solution

Most organizations are finding that corporate survival requires shifting course. By creating an EDA based on your organization's fundamental data building blocks, you'll provide a foundation on which to respond rapidly to the changing demands placed on your business in this millennium.

The shift from building departmental solutions to building enterprise solutions will require major cultural changes within the enterprise's business and IT departments. To compete and win in this century, you must discontinue building business systems and data stores that reflect management organization. Instead, focus on future business systems and data stores that reflect the true needs inherent in delivering products and services to the customer.

by Rand Losey

Information Management Magazine, January 2008

Logical data model reviews are an integral part of the modeling process. Periodic reviews of the logical data model allow the data architect to make minor adjustments rather than radical change. The number and frequency of reviews vary according to the size of the modeling effort. However, even the smallest modeling project should include at least two reviews, an initial review and a final review. For extensive modeling efforts that span several months, consider reviewing the logical data model at evenly spaced intervals, no more than three to four weeks apart.

The initial logical data model review should occur immediately upon completion of the first draft of the model. Conducting the review before the logical data model is fairly stable is a wasted effort because the model will likely experience substantial change. On the other hand, waiting until the logical data model is complete can be an expensive delay that might require extensive rework. Problems that could have been corrected with little effort early in the process might result in errors that compound into fatal flaws.

A review of the logical data model involves a comprehensive analysis of the entities, data elements and relationships in the model. The model is reviewed for completeness, standards compliance, clarity, reuse of existing designs and business partner understanding.

During the review process, defect severity levels should be assigned to each logical data model defect:

• Major – Major standards violation (project cannot continue unless and until defect is corrected).
• Minor – Minor standards violation (format, typographical).
• Comment – No impact (possible better way to same result).

Completeness and Standards Compliance Review

During the completeness and standards compliance review, the following questions should be asked. If a component of the logical data model fails a given question, a defect severity level should be assigned to the component/question combination.

• Does the logical data model include information pertinent to the business requirements and exclude nonpertinent information?
• Will all business questions be met by the logical data model?
• Is the logical data model in third normal form?
• Are all states of time (present, future, history) adequately reflected in the logical data model?
• Is there an IT activity or business process that will create, update, retrieve or delete each data element in the logical data model? If not, the data element is not necessary.
• Are the entities a person, place, thing (concept or event)?
• Are all entities named singularly?
• Is each entity fully described in the data dictionary?
• Is each entity named according to the naming standards?
• Is each entity defined according to the definition standards?
• Do all entities have a surrogate primary key?
• Is each data element fully described in the data dictionary?
• Is each data element named according to the naming standards?
• Is each data element defined according to the definition standards?
• Does each data element name have an approved and proper class word?
• Does each data element have an approved data type?
• Is each data element “self defining” (no other data is kept about it)?
• Do the relationship verb phrases (a.k.a. business rules) construct grammatically correct English sentences between the child and parent entities? E.g., “A Mortgage Product may be sold through one or more Parties.”
• Are all relationships important associations between occurrences of one or more entities that give information of value to the business?
• Is each relationship defined according to the definition standards?
• Is each relationship named according to the naming standards?
• Is each relationship fully described in the data dictionary?
• Do any outstanding completeness issues remain unresolved?

Clarity Review

During the clarity review, the following questions should be asked. If a component of the logical data model fails a given question, a defect severity level should be assigned to the component/question combination.

• Will any technical person reading the logical data model have the same understanding of its contents?
• Will any subject matter expert reading the logical data model have the same understanding of its contents?
• Will any business partner reading the logical data model have the same understanding of its contents?
• Are any of the entity names incorrect or misleading?
• Are any of the entity definitions incorrect or misleading?
• Are any of the data element names incorrect or misleading?
• Are any of the data element definitions incorrect or misleading?
• Are any of the relationship definitions incorrect or misleading?
• Are any of the relationship verb phrases incorrect or misleading?
• Could more information be added to improve the clarity of the logical data model?
• Could any of the information contained in the logical data model lead someone to an incorrect conclusion or misuse of the data?
• Do any outstanding “clarity” issues remain unresolved?

Reuse of Existing Designs Review

During the reuse of existing designs review, the following questions should be asked. If a component of the logical data model fails a given question, a defect severity level should be assigned to the component/question combination.

• Is the component under review based upon one of the logical data model reusable design patterns? Which one?
• Are you reusing existing enterprise logical data model entities? Which ones?
• Are all entities that reflect the same business requirement spelled, defined, and used in the same fashion?
• Are you reusing existing enterprise logical data model data elements? Which ones?
• Are all data elements that reflect the same business requirement spelled, defined and used in the same fashion?
• Are you reusing existing enterprise logical data model relationships? Which ones?
• Are all relationships that reflect the same business requirement spelled, defined and used in the same fashion?
• Do any outstanding reuse issues remain unresolved?

Business Partner Understanding Review

During the business partner understanding review, the following questions should be asked. If a component of the logical data model fails a given question, a defect severity level should be assigned to the component/question combination.

• Are all entity definitions meaningful and correct?
• Are all data element definitions meaningful and correct?
• Are all relationship definitions meaningful and correct?
• Do all relationship verb phrases represent business reality?
• Will any subject matter expert reading the logical data model have the same understanding of its contents?
• Will any business partner reading the logical data model have the same understanding of its contents?
• Could more information be added to improve the clarity of the logical data model?
• Could any of the information contained in the logical data model lead someone to an incorrect conclusion or misuse of the data?
• Does the business partner have the means and the will to collect the data for each data element?
• Does the logical data model include information pertinent to the business requirements and exclude non pertinent information?
• Will all business questions be met by the logical data model?
• Do any outstanding “business partner review” issues remain unresolved?

Business Partner Review Techniques

We must be cognizant of different communication techniques while reviewing the logical data model with the business partner or subject matter expert. Some trained business partners and subject matter experts may be skilled in reading a logical data model directly while others are more effective using other forms of documentation to fully understand its contents.

For those who are not skilled at reading a logical data model directly, the following business rules spreadsheet provides effective documentation. The spreadsheet provides the business partner with a concise and understandable way to communicate and validate the business rules that are represented in the logical data model as relationship verb phrases between child and parent entities. Relationship verb phrases (a.k.a. business rules) construct grammatically correct English sentences between the child and parent entities. For example, “A Mortgage Product may be sold through one or more Parties.”

 

Figure 1: Business Rules

With the business rules spreadsheet and data dictionary reports of all entities, data elements and relationships of interest to the business partner, you will be successful at validating your logical data model without forcing them to read the model directly.

Thriving in a Culture of Cooperation

A high-quality logical data model is built according to a recognized set of rules that enable the model to be used to greatest effect. The importance of these rules is that the organization can use their logical data models with assurance that they will be uniformly understandable and consistent regardless of who developed the model or when it was designed.

Integration is defined as the organization of all parts of the problem domain into a unified whole. Translated into a logical data modeling context, this definition means that all entities, data elements and relationships that reflect the same business requirement are spelled, defined and used in the same fashion.

Logical data models are used as a communication tool between the business community and the IT staff. Data requirements must be well documented and understood by all involved, must be business-oriented and must consist of an appropriate level of detail. When developing these models, the objectives must always be clarity and precision. When adding information to a logical data model, the data architect should ask whether the addition adds to clarity or subtracts from it.



by Rand Losey

Information Management Magazine, July 2015

From a technology perspective, creating a robust enterprise data warehouse (EDW) environment has proven to be a complex undertaking, but ultimately attainable.  The typical EDW initiative is driven by the business need for organizations to compete at a national or international level.  This requires data standardization, data conformance and data sharing.  An EDW sets the foundation for sharing data across the enterprise in a controlled and managed environment providing the end-user community with an integrated view of enterprise data to enable cross-functional and cross-departmental strategic and tactical decisions.

One of the primary reasons for EDW initiative failures is not technology related but is instead directly related to change resistance.  This should not surprise anyone given the aversion many of us have to change.  If we understand the reasons behind this fear of change, we will be better prepared to use our persuasion skills with those individuals who can do the most damage to our enterprise-wide data sharing goals. 

Blinded By Success

Your organization is not built on statistics - bottom lines, stock prices, global market forces - but on ideas and visions of what the business could be, what its leaders want it to become.  What prevents us from seeing into the future and embracing the changes necessary to enable these new visions?  Many leaders, especially those in successful businesses naturally tend to keep playing the same old tunes.  After all, why argue with success?  

The problem with this philosophy is that nothing on this earth is static.  Change is a fact of life; indeed survival is based upon adaptation to changing conditions.  Your organization is no different.  What if your industry was significantly deregulated?  You can bet that your forward-thinking competitors will be planning to prosper in a deregulated environment.  Some of your competitors have a sixth sense for industry change.  They know it is coming and they have an intuitive sense of how their business can reposition itself in order to take advantage of these changes.  We have to be careful that we are not prevented from seeing the future because we are blinded by the sum of our past and current successes.

One of the biggest challenges facing an organization when contemplating an EDW initiative occurs not in middle management, but at the senior management level.  Although middle managers appear to resist change, ultimately they will have to embrace change within your company or else find employment with another organization. 

When genuine disagreement occurs at the top, senior management often cannot agree on how to accomplish the goals of the EDW or even what the goals are.  Having gained personal power within the organization because they know how to effectively work within the current business model, senior managers can have heated disagreements with each other over the need to deploy an EDW.  Many executives are not comfortable sharing cross-functional or cross-departmental information with other senior managers.

In order to implement an EDW it is imperative to gain senior management commitment.  In many organizations, change agents face a great deal of resistance both inside the IT department and outside it when data integration projects are attempted.  IT professionals are by nature a creative lot, who enjoy the challenge of unraveling the nightmare world of data interfaces many organizations struggle under.  Outside, in user departments, the user wants exclusive rights over their own data.  Senior management must motivate both groups to get onboard the EDW project.  Senior management must directly participate in the planning and quality reviews of the EDW project.  Without this level of commitment, middle management may oppose and undermine the project.

Overcoming Resistance to Change

What is the secret to getting to a robust enterprise data warehouse (EDW) environment?  There is no secret at all; it essentially boils down to a few straightforward tactics:

• Give individuals a chance for involvement.  This can help them feel more committed to the EDW initiative.
• Communicate what is happening at every step of the EDW initiative.  Let your constituents know what they can anticipate next.  Information counts in building commitment to a change especially when done with a step-by-step approach. 
• Dividing a big change into a number of small steps can help make it less risky and threatening.
• It is not only important to provide information to build commitment to change, but also arrange the timing of the information's release.  Give people advance notice, a warning, and a chance to adjust their thinking.
• Help people save face by not making critical remarks toward the old way of doing things.  Instead, commitment to change is ensured when past actions are put in perspective - as the apparently correct thing to have done then, but now times are different.  This way people do not lose face for changing; the opposite occurs.  They look strong and flexible.  They have been honored for what they accomplished under the old conditions, even if it is now time to change.
• Look for and reward pioneers, innovators and early successes to serve as role models.
• Make sure people feel competent by ensuring there is sufficient training and education available so they understand what is happening.

Remember the News Years' resolutions you made on January 1st, about changing something in your life.  Typically, these resolutions, while well meant, are not totally implemented by us during the year.  This behavior is universal.  How many times have you said to yourself that I will lose 10 pounds this year? 

Deep down your obstructionist constituents want to change.  With your help they will change their mind setting the stage for the comprehensive enterprise-wide business intelligence and the extraordinary return on investment that an EDW environment can deliver to your organization.





by Rand Losey

Information Management Magazine, August 2015

A recent CIO survey showed that business intelligence (BI) applications were the highest technology priority. Of course, BI is a technology-driven process for analyzing data and presenting actionable information to help corporate executives, business managers and other end users make more informed business decisions.

BI encompasses a variety of tools, applications and methodologies that enable organizations to collect data from internal systems and external sources, prepare it for analysis, develop and run queries against the data, and create analytical reports for corporate decision makers as well as operational workers.

• No metrics established or captured.
• No sizing, estimating or planning tools.
• No progress reporting or design reviews.
• Bait and switch fixed bid contracts.
• No estimating heuristics captured from previous similar development efforts.
• No code inspection or defect tracking mechanisms in place.
• No formal change control (scope creep) mechanisms in place.
• Requirements which are unstable or unwritten.
• Excessive schedule pressures.
• Divisive politics between BI departments and the user community.
• Naive senior business executives who fail to oversee projects effectively.
• Geographically separate teams.
• Noisy open-plan offices which degrades productivity.
• Matrix management of projects – less efficient than hierarchical management where lines of control are clearer.
• Excessive amounts of staff attrition.
• Abrupt introductions of new technologies.
• Poorly trained project managers.
• Uncommitted senior BI and business managers – so the project ends up not being seen as important.
• Lack of ownership (personal accountability) issues – so it's no individual's fault if the project fails.
• Failures to integrate project goals within larger business aims.