Data Architecture – DataForecast

Here's a detailed briefing document summarizing the provided text on Data Architecture:

Briefing Document: Data Architecture

1. Introduction

This document provides a comprehensive overview of Data Architecture, drawing from the provided excerpts. It outlines the core concepts, activities, and implementation considerations, and addresses the increasing importance of data as an asset in modern organizations.

2. Core Concepts & Definitions

Architecture Defined: Architecture, in its broadest sense, refers to the organized arrangement of components to optimize function, performance, cost, and aesthetics. It is not just the end result but also the process of building.
Quote: "Architecture refers to the art and science of building things... and to the results of the process of building – the buildings themselves."
System Architecture: ISO/IEC 42010:2007 defines architecture as "the fundamental organization of a system, embodied in its components, their relationships to each other and the environment, and the principles governing its design and evolution.”
Data Architecture as a Discipline: Data architecture is a facet of information systems design that focuses on the organization, management, and use of data. It encompasses:
Artifacts: Models, definitions, data flows.
Activities: Forming, deploying, and fulfilling Data Architecture intentions.
Behavior: Collaboration, mindsets, and skills of various roles.
Levels of Architecture: Architecture practice occurs at various levels within an organization (enterprise, domain, project) and areas of focus (infrastructure, application, and data).
Enterprise Architecture: Encompasses domain architectures including business, data, application, and technology.
Data Architecture's Purpose: Data architecture helps organizations represent data at different levels of abstraction, enabling understanding and informed decision-making. It is fundamental to data management.
Data Architecture Artifacts: Specifications that describe the existing state, define data requirements, guide data integration, and control data assets. These are vital Metadata and should be stored in an architecture artifact repository.

3. Business Drivers

Bridge Between Strategy & Execution: Data Architecture acts as a bridge connecting business strategy to technology implementation.
Key Responsibilities of Data Architects:Evolving products and services based on emerging technologies
Translating business needs into data requirements
Managing data and information delivery
Facilitating alignment between Business and IT
Acting as agents for change, transformation, and agility
Value Creation: Data architects create and maintain organizational knowledge about data and the systems it moves through. This knowledge allows organizations to manage data as an asset and realize value through improved data usage, cost reduction, and risk mitigation.
Primary Outcomes:
Data storage and processing requirements
Designs of structures and plans that meet the current and long-term data requirements of the enterprise
Value Drivers: Achieving value for the organization by optimizing technical footprints, improving efficiencies, and increasing data usability.
Data Architect Specification Goals:Define the current state of data in the organization
Provide a standard business vocabulary for data and components
Align Data Architecture with enterprise strategy and business architecture
Express strategic data requirements
Outline high-level integrated designs
Integrate with the overall enterprise architecture roadmap.

4. Essential Concepts

Architecture Domains: Data Architecture operates within the context of other domains such as business, application, and technical architecture. Collaboration between these domains is crucial as they influence and constrain each other.
Enterprise Architecture Frameworks: Provide methods for understanding architecture. Common frameworks, like Zachman, include Data Architecture as an architectural domain.
Zachman Framework: A well-known enterprise architecture framework that uses a 6x6 matrix to show the different perspectives on architecture, recognizing that different audiences have differing views.
Two Dimensions:Communication Interrogatives (Columns): What, how, where, who, when, and why.
Reification Transformations (Rows): Identification, definition, representation, specification, configuration, and instantiation. (Corresponding to: planner, owner, designer, builder, implementer, user)
Cells: Each cell represents a unique type of design artifact resulting from the intersection of its row and column.
Enterprise Data Architecture (EDA): Defines standards and designs for important organizational elements, including data collection, storage, integration, movement, and distribution.
Components of EDA:Enterprise Data Model (EDM): A holistic, implementation-independent view of data including entities, relationships, business rules, and critical attributes. Serves as the foundation for data projects.
Data Flow Design: Defines how data moves through various components like databases, applications, and networks, linking data to business processes, roles, locations, and technical components.

5. Enterprise Data Model (EDM) Deep Dive

Stand-Alone Artifact or Composed of Models: An EDM can be created as a stand-alone artifact or understood as a collection of models that describe the data entities, attributes, and relationships.
Industry Standards: Adopting an industry-standard model can jumpstart the process, but enterprise-wide data models require significant investment to tailor them to specific organizational needs.
Incremental Approach: Most successful EDMs are built incrementally and iteratively using layers.
Levels of an EDM:Conceptual overview of subject areas.
Views of entities and relationships for each subject area.
Detailed, partially attributed logical views of the same subject areas.
Logical and physical models specific to an application or project.
Relationships Between ModelsVertical: Models at each level map to those at other levels, creating lineage.
Horizontal: Entities and relationships can exist in multiple models within the same level and can be linked across subject areas.
Subject Area Models: The conceptual enterprise data model is built up by the combination of subject area models. These subject areas must be consistent.
Subject Area Discriminators: Subject area structures are usually formed with normalization rules for most effective Data Architecture work. Other principles for forming subject area structures include funding, governance structures, top-level processes, and business capabilities.

6. Data Flow Design Deep Dive

Data Lineage Documentation: Data flows document how data moves through business processes and systems.
Data Flow Elements: Data flows illustrate where data originated, where it is stored, how it is used, and how it is transformed. Data lineage analysis helps explain the state of data at any point in the flow.
Data Flow Mappings: Data flows map relationships between data and:
Applications within a business process
Data stores or databases
Network segments
Business roles (CRUD)
Locations where local differences occur
Levels of Detail: Data flows can be documented at subject area, business entity, or attribute level. Systems can be represented by network segments, platforms, or individual servers.
Representations: Data flows can be depicted by 2-dimensional matrices or data flow diagrams. Matrices show data created and used, while diagrams illustrate data flow between systems.

7. Activities

Complexity Management: Data and enterprise architecture address complexity from two angles:
Quality-Oriented: Focus on improving execution within business and IT development cycles, maintaining architectural quality, and improving standardization.
Innovation-Oriented: Focus on transforming business and IT to leverage new technologies and opportunities and using leading-edge technologies. This approach requires architects to engage with diverse roles within the organization
Establishing Data Architecture Practice:Data Architecture should be an integral part of enterprise architecture.
Use a framework to articulate goals, drivers, and scope for Data Architecture and communication with stakeholders.
Develop work streams that include strategy, acceptance/culture, organization, working methods, and results.
Impact on Projects and System Releases: Enterprise Data Architecture influences:
Project data requirements
Review of project data designs to ensure consistency
Determination of data lineage impacts
Data replication control
Enforcement of Data Architecture standards
Guidance of data technology decisions.
Evaluate Existing Specs: Identify existing system documentation, evaluate, and update as necessary.
Develop a Roadmap: Provides a plan for a 3-5 year development path to manage data dependencies and tradeoffs. The roadmap should be integrated with the overall enterprise architecture roadmap.
Data Dependencies: Should resolve the business capability dependencies in the order that data is created.
Managing Enterprise Requirements Within Projects:Ensure data models and specifications are flexible to accommodate future needs.
Architects should analyze project specifications for adherence to standards, inclusion in the overall EDA, generalizability, and new delivery architectures.
Address Data Architecture early in project planning.
Project activities: define scope, understand business requirements, design detailed specifications, implement.
Integration with Development Methodologies:Waterfall: Include Data Architecture activities in sequential phases.
Incremental: Create a comprehensive data design in early iterations.
Agile: Include specifications for data models, capture, storage, and distribution. Strong collaboration between programmers and data architects is vital.
Integrating with Enterprise Architecture: EDA work is driven by projects, but Enterprise-wide EDA needs to be proactive and integrate with project portfolio management, and enterprise application development and integration planning.

8. Tools

Data Modeling Tools: Necessary for managing the EDM at all levels, including lineage and relationship tracking.
Asset Management Software: Inventories systems, describes content, tracks relationships and provides valuable Metadata to inform data flows.
Graphical Design Applications: Used to create architectural design diagrams, data flows, value chains and other artifacts.

9. Techniques

Lifecycle Projections: Categorize architecture designs based on current use, future deployment, retirement plans, and use policies.
Diagramming Clarity: Use consistent visual conventions to avoid misinterpretations, including clear legends, line direction, consistent line cross displays, consistent object attributes, and linear symmetry.

10. Implementation Guidelines

Key Aspects of Implementation:Organizing Data Architecture teams and forums.
Producing initial versions of artifacts (EDM, data flow, roadmaps).
Establishing a Data Architecture way of working in development projects.
Raising organizational awareness of the value of Data Architecture.
Incremental Implementation: Begin in part of organization or data domain and grow wider after learning and maturing. Early projects often require larger portions of Data Architecture work.
Business Drivers: Influence the implementation strategy, whether it's an agile approach in a culture that values innovation or a more structured approach in a quality-driven culture.
Traditional Approach: Start with master data improvements, then expand to transactional data and use governance to ensure compliance.

11. Readiness / Risk Assessment

Significant Risks:Lack of management support.
No proven record of accomplishment.
Apprehensive sponsor.
Counter-productive executive decisions.
Culture shock.
Inexperienced project leader.
Dominance of a one-dimensional view.

12. Organization and Cultural Change

Culture's Impact: Architectural adoption speed depends on the adaptability of the culture. Data architects must collaborate with creative thinkers, who might embrace or resist the changes.
Ideal Organizations: Output-oriented, strategically aligned organizations are best positioned to adopt architectural practices as they prioritize based on common objectives.
Factors for Architectural Adoption:
Cultural receptivity to architecture.
Recognition of data as a business asset.
Ability to adopt an enterprise perspective.
Ability to integrate architectural deliverables into project methodologies.
Acceptance of formal data governance.
Holistic view of the enterprise.

13. Data Architecture Governance

Data Architecture's Role: Supports the alignment and control of data and acts as a liaison with data governance.
Governance Alignment: Enterprise Data Architecture should be aligned with Data Governance organization. Data Architect and Data Steward should be assigned to subject areas or entities, along with business oversight aligned with process oversight.
Activities:
Overseeing projects for compliance with architectural standards.
Managing designs, lifecycle, and tools.
Setting rules, guidelines, and specifications for data use.
Creating compliance artifacts.

14. Metrics

Performance Metrics: Reflect architectural goals: compliance, implementation trends, and business value. Tracked annually as part of overall project customer satisfaction.
Architecture Standard Compliance Rate: Measures how closely projects follow standards and engage with architecture.
Implementation Trends: Track how enterprise architecture improves project implementation, reuse of artifacts, and project efficiency.
Business Value Measurements: Track progress toward business objectives and benefits:
Business agility improvements
Business quality measurements
Business operation quality measurements
Business environment improvements

15. Works Cited / Recommended * A list of books on Data and Enterprise Architecture was provided.

Conclusion

This briefing document provides a thorough review of Data Architecture, covering its definition, business drivers, key concepts, activities, implementation, and governance. It underscores the critical role Data Architecture plays in enabling organizations to effectively manage and leverage data as a strategic asset.