The Open Group is a global consortium that enables the achievement of business objectives through technology standards. Our diverse membership of more than 700 organizations includes customers, systems and solutions suppliers, tools vendors, integrators, academics, and consultants across multiple industries.

The mission of The Open Group is to drive the creation of Boundaryless Information Flow™ achieved by:

Further information on The Open Group is available at www.opengroup.org.

The Open Group publishes a wide range of technical documentation, most of which is focused on development of Standards and Guides, but which also includes white papers, technical studies, certification and testing documentation, and business titles. Full details and a catalog are available at www.opengroup.org/library.

This document is a Snapshot of what is intended to become The Open Group Agile Architecture Framework™ Standard, also known as the O-AAF™ Standard. It is being developed by The Open Group.

This document follows a modular structure and is organized in four parts:

This first Snapshot document has fully developed an initial release of the architecture fundamentals section. The other sections are still incomplete and will be completed in the next version.

The target audience for this document includes:

This document does not contain any normative references at the time of publication. These may be added in a future release.

This Snapshot document is a draft of The Open Group Agile Architecture Framework™ Standard. The objective of this document is to cover both Digital Transformation of the enterprise, together with Agile Transformation of the enterprise.

This Snapshot document is intended to make public the direction and thinking about the path we are taking in the development of The Open Group Agile Architecture Framework Standard. We invite your feedback and guidance. To provide feedback on this document, please send comments by email to ogspecs-snapshot-feedback@opengroup.org no later than January 15, 2020.

This Snapshot documents a proposal for a standard that covers both Digital Transformation of the enterprise, together with Agile Transformation of the enterprise. The scope of the Snapshot covers key concepts and includes the topics below:

Additional "how-to" architecture material will be developed in the next version.

This is a Snapshot, not an approved standard. Do not specify or claim conformance to it.

The following standards contain provisions which, through references in this standard, constitute provisions of the O-AAF Standard. At the time of publication, the editions indicated were valid.

For the purposes of this document, the following terminology definitions apply:

The Open Group intends to progress this activity towards publication of a standard of The Open Group.

The Why? questions start a strategy formulation process composed of several building blocks [McKinsey 2013]:

Strategy evolves from big up-front design toward a learning process that involves all organizational levels. The strategic process itself must be adaptive and capable of responding to changes in the business system. The people who are charged with executing the strategic plan should participate in the planning process itself [Watson 2005].

The Japanese have coined the term catchball which refers to a strategic planning approach that incorporates structured group dialog at all levels of the organization. The key benefits of the catchball process are to drive alignment among all silos and hierarchical levels, and to help shape the business as a coherent whole aligned on its core objectives.

This view of strategy formulation and deployment goes hand-in-hand with a flatter and cross-functional organization. It also requires the enterprise’s governance model to evolve. Agile teams are granted more autonomy but not at the expense of effective alignment. The catchball process provides an alignment mechanism that is compatible with autonomy because it works best in a non-control and command culture.

The "acid" test of a successful business strategy is the increased capability of the enterprise to develop offerings that customers are willing to pay for and that deliver the right experience.

In his book “Think like Amazon™” [Rossman 2019], John Rossman writes that digital is about two things: "speed and agility – externally to your customers and market and internally within your organization".

Team autonomy is a prerequisite to speed and agility. Why? If the coordination work your teams have to perform is too high, it will slow them down, which results in a slower pace and additional rigidity.

John Rossman claims that speed is: "about moving in one direction very efficiently, very precisely. Operational excellence at scale is the business equivalent to speed".

Operational excellence implies effective coordination between autonomous teams. Freedom needs to be combined with responsibility. Accountability relations that link teams bring predictability. For example, at Amazon you are expected to identify and tenaciously manage every potential business-derailing dependency you have. Peer pressure is more powerful than pressure from managers who do not have the bandwidth to micro-manage inter-team dependencies.

In today’s digital world, the quality of customer experience is becoming a critical success factor. Consumers and B2B customers have expectations that are shaped by the Internet giants. For example, the level of service provided by Amazon Prime™ is becoming the new normal of the retail industry. Because most Amazon competitors do not have the resources and scale to provide a similar service, the logistic capability becomes a decisive competitive advantage.

This has forced large retailers such as Walmart® to raise their investment level to develop new digital capabilities on par with those of Amazon. Recently a startup has developed a service to provide state-of-the-art logistic services to smaller retailers which do not have the skills nor the scale to compete with Amazon’s logistic capabilities.

Amazon, with the purchase of Whole Foods, adds stores to its distribution strategy. The future of retail may be "click and mortar" executed in the context of an omnichannel business model.

Customer experience starts with the discovery of customer needs. Just asking clients what they need is not going to help create a superior customer experience. Discovering the hidden or untold needs of clients is key to success. The analysis of the customer’s job-to-be-done or the mapping of the customer journey are two examples of practices on which digital enterprises are relying. The rapid adoption of design thinking is helping enterprises better understand the spoken and unspoken needs of their clients.

Design thinking is a human-centered approach that positions empathy as the centerpiece of the design discipline. It provides a process that helps design better products, services, or processes. The design thinking process has five steps: Empathize, Define, Ideate, Prototype, and Test, which are not intended to be applied in a linear manner.

Once the problem space is well understood, the enterprise can architect a product and service system that will satisfy its clients. Though the product concept is central to Agile, the Agile literature gives few if any product definitions.

The terms "product" and "service" have different meanings depending of their context of use. The common meanings of product are:

This document defines a product as something a value stream produces. A product has a lifecycle which is comprised of a product and process development value stream and a production value stream. Broadly speaking a product can refer to a product or a service. A service will be referred to as a product if its delivery is industrialized or repeatable.

In an Agile context, product-centricity refers to the shift from temporary organizational structures – projects – to permanent ones. A product-centric organization is composed of cross-functional Agile teams which are responsible for developing products or services and operating or running them. The DevOps principle "you build it, you run it" is core to product-centricity.

When digging into the subtleties of the various product and service meanings a few key ideas emerge:

Using the word "offering" helps talk about product, service, or a combination in a generic way. This avoids some of the semantic problems linked to the variety of meanings carried by the product and service terminology.

The following definition borrowed from the Lean body of knowledge summarizes how the ideas above can be combined to guide the design of digital offerings: "A product is an object, material or digital, that allows its owner to solve a problem autonomously. There are always limits to that of course, as a product will need consumables, services, and maybe even training to use it more efficiently. But the ideal product is one that I use effortlessly, grasping intuitively how it works without needing to talk to anyone else to make it work or maintain it. I’m autonomous." [Ballé 2019]

Agile shifts from project-oriented management to product-oriented management. A true Agile team does not deliver a project but a product! Product-centricity drives the Agile organization: "Moving from project-oriented management to product-oriented management is critical for connecting software delivery to the business" [Kersten 2018]. The product-based focus helps create stable Agile teams that develop an end-to-end perspective and are staffed with stable resources.

The Agile enterprise architects its product and service system in a modular way to minimize inter-team dependencies. The lower the dependencies, the faster products are delivered to the market. Capability modeling and Domain Driven Design (DDD) help guide the modular decomposition of the enterprise and its software systems.

The Inverse Conway Manoeuvre suggests modeling Agile teams' structure to map the intentional system architecture’s structure. When the teams' architecture mirrors the software system’s architecture, it reinforces the development of an end-to-end perspective that improves effectiveness and efficiency.

The term "digital offerings" denotes solutions that bundle products and services that help customers do their jobs. Time-to-market as well as quality are improved when the enterprise creates digital platforms that are made of reusable software components. The development of innovative business models helps enterprises differentiate their value proposition. Competing on business model is more effective than just competing on products.

The clear distinction between problem space and solution space facilitates innovation because it opens the space of possible solutions. More than one solution can solve a problem that has been well formulated.

A complex system has many elements or entities that are highly interrelated, interconnected, or interwoven [Crawley 2016].

John Holland [Holland 2014] characterizes the behavior of complex systems by:

When interaction patterns emerge, it influences the behavior of interacting agents and therefore impacts the system itself. A complex system has emergent properties that cannot be understood nor predicted by just breaking it down into its elements. This lack of predictability makes steering the evolution of complex systems difficult. A complex system can evolve toward self-organization or chaos should it fail to maintain some kind of invariant state in the face of perturbations.

An Agile at scale enterprise that runs hundreds of Agile teams in parallel is a complex system. A fine line must be walked between self-organization and chaos. A research paper from MIT Sloan describes how Spotify™ balances autonomy with alignment [Baiyere 2017]. Spotify avoids chaos while it does not slow down innovation. The definition of shared missions which are key strategic objectives helps align Agile teams.

John Holland suggests steering complex systems by modifying their signal/boundary hierarchies. The components of complex systems are bounded sub-systems or agents that adapt or learn when they interact. Defining the boundary of sub-systems and their rules of interaction has a major influence on the evolution of a system. For example, establishing a taxonomy of Agile teams and clarifying their dependencies impact coordination and cooperation. Similarly, Domain-Driven Design (DDD) uses the bounded context concept to decompose complex software systems in a modular way [Evans 2003].

Ed Crawley defines architecture as: "the embodiment of concept, and the allocation of physical/informational functions (processes) to elements of form (objects) and definition of structural interfaces among the objects".

It consists of:

Form is what a system is; it is the physical or informational embodiment that exists or has the potential to exist. Form has shape, configuration, arrangement, or layout. Over some period of time, form keeps its identity though it can change state. Form is not function, but form is necessary to deliver function. For example, in a software system, form defines the sub-systems and entities that compose it. In an enterprise, form defines the organization and the products that the organization produces.

Function is what a system does; it is the activities, operations, and transformations that cause, create, or contribute to performance. For example, the function of an Agile team (e.g., a tribe) could be to develop a payment product and the sharding function is to distribute data over a network. Emergence occurs in the functional domain.

A concept is a product or system vision, idea, or mental image which maps function to form. In the sharding example, the distributed computing concept of master-slave replication defines how to distribute data while keeping it consistent.

Upstream Influences

The design of an architecture is influenced by many factors such as the enterprise’s business strategy, compliance requirements, the needs of clients and users, or the behavior of competitors. When these upstream influences are not dealt with, ambiguity is likely to jeopardize the design of the new product or system. One of the key architecture missions is to help reduce this ambiguity.

A well-architected system delivers value to its stakeholders within the limits imposed by regulators, competition, and technology.

Value and Architecture

As defined by L.D. Miles from General Electric®, value is the lowest price you must pay to provide a reliable function or service. Value is the ratio of function to cost.

The value of a product can be increased by maintaining its functions and reducing its cost or keeping the cost constant and increasing the functionality of the product.

A good architecture delivers expected benefits at competitive cost.

Downstream Influences

Downstream influences extend the architecture’s scope to the full product or system lifecycle. For example, product architecture decisions ought to factor in manufacturing costs or ease-of-maintenance of products or systems. When architecting a product, a product line strategy can influence architecture decisions.

ISO/IEC/IEEE 42010:2011 (https://www.iso.org/standard/50508.html) defines "architecture" as: "The fundamental concepts or properties of a system in its environment embodied in its elements, relationships, and in the principles of its design and evolution."

The emphasis is on form elements and relationships, and the definition does not give much guidance on function and concept.

The TOGAF standard embraces the ISO definition, but does not strictly adhere to it, adding the definition below:

"The structure of components, their inter-relationships, and the principles and guidelines governing their design and evolution over time."

The emphasis on form is expressed through a different vocabulary: the "structure of components and relationships" versus "elements and relationships".

The system engineering definition of architecture used in this document is fully compatible with the ISO and TOGAF definitions.

In an Agile world, products and systems evolve incrementally through rapid learning cycles. How stable an architecture remains during these iterations can translate into two questions:

Chapter 5, Continuous Architectural Refactoring will describe how to design architectures that can evolve gracefully. It will explain the role played by refactoring to help meet the evolvability’s architecture quality.

A point to note when answering the first question is: what is the right balance between anticipating foreseeable change versus over engineering? Not all change can or should be anticipated.

There are cases when it is wise to make a product or system resilient to change. For example, using a strategy pattern that enables selecting a pricing algorithm at run time instead of implementing it at design time is useful when designing a trading system.

The second question raises the issue of the reversibility of architecture decisions. Hard to reverse architecture decisions make incremental architecture change more difficult. Chapter 9, Minimum Viable Architecture will cover this topic in more depth.

In this section we discuss the topic of Continuous Architectural Refactoring, a topic of increasing importance in today’s Agile, DevOps-oriented software landscape.

However, to properly discuss the concept, we must first adequately define it, and every word in the term is important. We will briefly discuss each in turn (but not in order).

The remainder of this section will discuss the key considerations in planning for continuous architectural refactoring of a software system; it answers the question of how we can set ourselves up to be in a position to continuously evolve our architectures in response to changing requirements, architectural debt, and other headwinds. We detail these under three headings:

Each sub-section covers a different aspect of the necessary prerequisites for continuous architectural refactoring. Taken together they offer a complete view of the enablers for successful continuous architectural refactoring.

Before we can decide upon what technical and organizational mechanisms can be put in place to facilitate continuous refactoring, we must first understand the conditions under which we are operating. Once we have identified the business and technical constraints relevant to our system, we can then put in place structures that will allow us to evolve within those constraints. Fitness functions will allow us to actually test that our architecture is fit-for-purpose. Guardrails will contribute to the guidance referred to in the definition of architecture in Ford [Ford 2017], keeping our development teams from going astray in their system designs.

Successful continuous architectural refactoring needs the development team to be empowered to iteratively make architectural changes. There are a number of key technical enablers for this which are discussed here:

In addition, Agile development practices are a key enabler for continuous architectural refactoring. As described in Chapter 6, Architecting the Digital Enterprise, there are a number of practices which are promoted by Agile working. These practices allow continuous architectural refactoring to be successfully implemented; in particular the rapid iteration and experimentation, which allows architectural evolution to be readily incorporated into ongoing development activities.

Technical mechanisms such as continuous delivery and feature toggles are powerful enablers of continuous architectural refactoring, but they are certainly not the only ones. For example, what if you didn’t have the buy-in of senior management to do any refactoring? (Hint: architectural refactoring gets continuously prioritized behind functional evolution.) Even if you have such buy-in, to paraphrase the definition of architecture in Ford [Ford 2017], continuous refactoring needs to be guided and incremental. The guidance comes in the form of an architectural roadmap, a best-guess hypothesis of how the architecture needs to evolve. Finally, organizations need to balance the tensions between these forces; sometimes we should refactor; sometimes we should build new functionality.

Before we continue, it is worth noting that development team structure is also a key enabler for continuous architectural refactoring, in particular the Inverse Conway Manoeuvre. This technique has been described separately in Chapter 3, A Dual Transformation.

Jean-Jacques Lambin [Lambin 2000] states: "the role of strategic marketing is to lead the firm towards attractive economic opportunities; that is, opportunities that are adapted to its resources and know-how and offer a potential for growth and profitability".

Strategic marketing is about:

"A positioning statement defines the value proposition of products to the target: …​ the point of difference (reason to buy) and the point of parity (point of reference)" [Wharton 2018].

Strategic marketing provides context to help drive digital strategy. The enterprise can choose to develop new capabilities to serve targeted client segments. Traditional strategic marketing is mostly developed in a top-down manner.

"Business model" is one of the great digital buzzwords. A good business model tells a good story. Describing an existing business model is easy and many templates can help you structure a story line; for example, the Business Model Canvas [Osterwalder 2010] or "Reinventing your Business Model" [Johnson 2008].

The difficult part is changing an existing business model or creating a brand new one that works. Following a generic template is no guarantee of success.

Successful business models start with an innovative value proposition that has been field tested. The Amazon Flywheel illustrates how systems thinking can help design successful business models. "Simply put, a flywheel is a self-reinforcing loop or systems diagram driven by key objectives or initiatives" [Rossman 2019].

John Rossman’s Idea 25 states: "Study and analyze either your industry or the situation you are trying to improve using systems thinking. Once you have an idea or hypothesis on how to achieve your goal, create a simple version of your system, often called a “flywheel”, to assist in testing your strategy and then in communicating your logic and plan to others".

Digital enterprises can pursue both differentiation and low cost. This creates a leap in value for both the enterprise and customers. The Amazon Prime Now™ service epitomizes this. Customers can purchase a large variety of products at a competitive price and get a free two-hour delivery at their home (https://primenow.amazon.com).

Product variety is increased through a platform that allows third-party vendors to sell and deliver their products leveraging Amazon’s portal and logistics capabilities. The Amazon business model is characterized by:

The Amazon example illustrates how firms can compete leveraging difficult-to-replicate capabilities which are enabled by a digital platform that enables an adaptive operating model.

Though classical marketing research provides a set of quantitative and qualitative tools to analyze customer needs and test product ideas, it is no guarantee of success. Therefore, the design discipline is gaining traction to gather customer insights which complement traditional marketing analysis.

Design thinking postulates that to create meaningful innovations, the enterprise needs to know its customers and care about their lives [Stanford 2010].

The concept of job-to-be-done is a key tool, developed by Clayton Christensen. It helps better understand customer needs by wrapping offered products and services in a usage context.

"After decades of watching great companies fail, we’ve come to the conclusion that the focus on correlation – and on knowing more and more about customers – is taking firms in the wrong direction. What they really need to home in on is the progress that the customer is trying to make in a given circumstance – what the customer hopes to accomplish. This is what we’ve come to call the job-to-be-done."

New market segmentation methods emerge from analyzing jobs-to-be-done. Customers can now be classified based on the outcomes they expect. Unlike abstract client segments, personas help better understand who the persons are the enterprise targets and what are their pains and expected gains.

The four remaining steps of design thinking help the enterprise formulate a better problem statement (define), generate a broad range of ideas (ideate), and prototype and test candidate solutions.

Validated customer insights provide key inputs to help define innovative value propositions.

Customer journey mapping helps to systematically put the customer at the heart of the enterprise’s digital strategy. It offers a new way of seeing the enterprise’s markets and strategy from an outside-in rather than an inside-out perspective. It helps teams and management "walk in the shoes" of their customers. It helps correct strategy myopia because it shows a much broader picture of customer needs than classical marketing addresses.

Customer journey maps come in many shapes and forms. For example, customer maps can follow a timeline positioning activities or phases chronologically.

Alternatively, they can be represented as:

In addition to describing the customer’s functional job, a customer journey map captures the feelings of customers during moments of truth. It can also capture how a customer believes she is perceived socially.

Internet giants are dominating competition by leveraging the power of the platform. A platform leverages technology to connect participants to create or exchange value. For example, Airbnb™ connects travelers with local hosts who earn extra income, or Uber™ connects interested passengers with drivers who use their own car to provide transportation services.

Platform-based business models are based on the two-sided markets theory developed by the Economics Nobel prizewinner Jean Tirole. When platform-based businesses enter markets dominated by "pipelines", they enjoy a competitive advantage. Why? Because pipelines rely on inefficient gatekeepers to manage the flow of value when platforms promote self-service and direct interactions between participants.

A platform can scale and grow more rapidly and efficiently because the traditional gatekeeper is replaced by signals provided by market participants through a platform that acts as a mediator. Platforms stimulate growth because they expose new supply and unlock new demand. They also use big/fast data and analytics capabilities to create community feedback loops [Parker 2016].

Technology is a key enabler of platform-based business models because high-levels of automation and self-service capabilities are required to succeed. For example, when Amazon decided to develop a third-party sellers' market, one of the key requirements was: "A third-party seller, in the middle of the night without talking to anyone, would be able to register, list an item, fulfill an order, and delight a customer as though Amazon the retailer had received the order.” [Rossman 2019]

This document makes a clear distinction between the platform business model which is based on the two-sided market theory and the digital platform which is a technology enabler that allows enterprises to achieve economic gains by reusing or redeploying assets across families of products.

A digital platform is a software system composed of application and infrastructure components that can be rapidly reconfigured using:

The definition in this document is compatible with Jeanne Ross' definition from the MIT CISR which defines a Digital Platform as: "a repository of business, data, and infrastructure components used to rapidly configure digital offerings" [Ross 2019].

Most established enterprises have monolithic legacy systems that get in the way of creating effective and efficient digital platforms. To compete with Internet giants or more nimble competitors, these enterprises must develop refactoring and modernization strategies.

True Digital Transformation means allowing the rapid creation and market testing of new digital offerings inspired by customer insights and powered by a digital platform to deliver differentiated outcomes to customers. "A digital offering is the confluence of a customer solution and a great experience" says Jeanne Ross [Ross 2018].

The most advanced enterprises use a Lean product development approach to create digital offerings [Oosterval 2010] [Morgan 2019].

The Lean Startup best practices are helping to market test digital offerings before enterprises invest too much in them [Ries 2011].

Delivering superior customer experience cannot be achieved by improving the front-end alone. If the back-end is the source of rigidity or defects, it will translate into poor customer experience. The customer journey should be driven by the customer and not by rigid back-end processes.

Figure 5, “Revised Service Blueprint”, inspired by service blueprinting, helps bridge customer journeys with required capabilities. The example describes a simplified loan origination journey. Variants of this journey can be created to account for a different channel mix.

The top part of the diagram borrows from a journey map. The bottom part below the line-of-visibility describes the capabilities that are required to support the story map. Capabilities are implemented by services that are described in functional terms and/or specified using APIs and/or business events.

Architecting a set of modular and reusable services will allow for rapid reconfiguration of customer journeys. System architecting techniques combined with Domain-Driven Design (DDD) can help design loosely-coupled services that will be easier to assemble into unanticipated composite services. An adaptive operating model takes advantage of modularity and composability to gracefully adapt to changing customer experience requirements.

In a paper published in the Sloan Management Review Jeanne Ross suggests to "initiate change by assigning accountabilities for specific business outcomes to small teams or individual problem owners" [Ross 2018].

In many enterprises this change impacts the formal organizational structure. For example, several banks are adopting the Spotify model to re-organize the IT function and some such as ING or ANZ are also re-organizing the business the same way.

Changing the organizational structure is not enough. It is key to change the culture, the ways of working, and the management system. Accountability is a powerful alignment mechanism when teams trust each other. Trust requires visibility and predictability which are not the hallmark of "command and control" organizations.

In the Big Design Up-front (BDUF) style, architecture teams often pick fundamental concepts based on incomplete or old knowledge. The subsequent design iterations:

The resulting solution is likely to require significant rework and will be suboptimal because good concepts are eliminated too early.

In contrast, Set-Based Concurrent Engineering (SBCE) [Ward 2014]:

"Taking time up-front to explore and document feasible solutions from design and manufacturing perspectives leads to tremendous gains in efficiency and product integration later in the process and for subsequent development cycles" [Sobek 1999].

In our context concurrent engineering brings agility and facilitates innovation, for example:

Figure 7, “Transformation Increment” shows that each transformation increment should cover the entire hierarchical line. Why? Because if you change the way one level of the organization operates while the hierarchical line continues to operate the old way, the enterprise runs the risk of submitting its employees and managers to a double bind; i.e., employees and managers becoming at risk of being confronted with two irreconcilable demands or a choice between two undesirable courses of action. For example, re-prioritize your backlog but stick to established budgets and product plans.

Incremental transformation deployment is easier when two conditions are met:

When those conditions are not met, it is better to change the organizational structure before deploying new ways of working and the new management system. The new organizational structure reflects an architectural intent which is either implicit or explicit.

We will illustrate the need to re-architect the organization with the example of an IT organization that evolved toward more agility.

The legacy IT organization was structured by process/function (front-office, middle-office, finance, risk) rather than by product families (plain vanilla swaps, equity derivative, commodities, etc.). Two powerful central functions would control operational teams:

Pools of software development resources were organized by technologies; for example, server-side Java® development or mobile front-end development. Software development projects would on-board required resources for the duration of the project and release them when done. Last but not the least, IT operations and support was separate from application development.

Though the specialization of the legacy organization was supposed to help the IT organization capture economies of skills and scale, several problems surfaced.

The level of inter-team dependency is high because of the multiplication of organizational entities that have to intervene on a project. Time-to-market is increasing due to the high level of inter-silo coordination required; for example, between development and IT operations teams. Alignment between business and IT is complicated because of the lack of a simple mapping between organizational structures.

The IT re-organization was inspired by the Spotify model. Small cross-functional teams named "squads" replaced projects. The size of a squad does not exceed 10 to 15 people. Unlike a project, a squad is a stable team composed of dedicated resources that develops an end-to-end perspective. Squads are product-centric, meaning they develop, deploy, and operate a service. Squads adopt a DevOps culture which translates into the mantra "you build it, you run it".

Squads are grouped into tribes which are not bigger, on average, than 150 people. In order to maintain and develop functional or technical competencies, chapters and guilds are created. For example, chapters that regroup mobile developers or NoSQL DBMS experts.

As the number of Agile teams grows with a few hundred squads running in parallel, it is important to define a taxonomy of Agile teams that clearly defines the scope of each one and minimizes inter-team dependencies. True Enterprise Architecture thinking is required to discover an effective way to decompose the organization and to draw boundaries that minimize inter-team dependencies.

Figure 8, “Agile Teams' Taxonomy” represents a simplified taxonomy.

The primary goal of an Agile teams' taxonomy is to minimize redundancy and duplication [Rigby 2018]. Because an Agile teams' taxonomy may be different from the formal P&L structure of the enterprise, it is necessary to map it to the division, business unit, and P&L structure of the enterprise.

Figure 9, “Maturity Levels” lists the CMMI V2.0 maturity levels. We like the idea of differentiating organization-wide practices from project-level practices.

Because Agile at scale shifts from project to product and organization-wide practices are not limited to standards, we need to define specific maturity levels.

Figure 10, “O-AAF Maturity Levels” introduces the maturity levels we propose.

For each maturity level, the enterprise should develop specific architecture practices.

Figure 11, “Architecture Practices” describes the essential architecture practices enterprises need to implement for each level.

We will now identify the practice areas that need to be analyzed for each maturity level.

Figure 12, “Practice Areas” defines six practice areas. For each area and each level, the matrix defines assessment criteria. Maturity levels in the table start at level 2 because levels 0 and 1 are out of scope of this document.

The practice areas table above can be used to assess the maturity level of an Agile enterprise. In the next version of this document, we will develop a maturity assessment method which will be described in a playbook.

Chapter 4, What is Architecture? explores what distinguishes architecture from design. Not all design decisions are architecture decisions. Architecture decisions need to be organized driven by the identification of high-impact decisions. High-impact decisions are defined as those which are likely to have significant effect on quality metrics or on other decisions. There is a need for guidelines on how best to organize the architecture decision space.

Wording such as building architecture or architecture runway may give the impression that architecture means infrastructure or platform. Infrastructures and platforms implement architecture models but should not be confused with architecture which is about the fundamental concepts and properties of a system. Therefore, we suggest distinguishing two types of decisions: the true architecture ones from infrastructure sourcing and provisioning ones.

Taking the example of an MVP, we suggest distinguishing architecture decisions that pre-condition its development, from investment decisions to fund its development and production environments.

An enterprise can be modeled as a complex system. A large enterprise may be composed of many divisions and departments; it can operate in many regions of the world and it can market a large number of products and services. When Agile at scale is deployed, the minimum architecture is the one required to define an Agile team’s taxonomy as described in Chapter 7, Architecting the Agile Transformation.

In an Agile culture where team autonomy is valued, architecture is the result of a problem-solving process that starts from an intentional architecture vision which challenged, amended, and completed by Agile teams.

What is the minimum definition of this intentional architecture vision? How to conduct the dialog between the owner of the architecture vision and Agile teams? It depends on the context. The heuristics described in the next section can help architects answer these questions.

This document proposes a set of heuristics to structure and time the architecture decisions which are contingent to the context in which they are made. The heuristics below can be combined.

Focus on Type 1 Decisions

Jeff Bezos, CEO of Amazon, distinguishes two types of decision: type 1 and type 2. Type 2 decisions are changeable and reversible; they are two-way doors. If you’ve made a suboptimal type 2 decision, you don’t have to live with the consequences for that long. Type 2 decisions can and should be made quickly by high judgment individuals or small groups." Type 2 decisions can be made by autonomous Agile teams, while type 1 decisions require architecture thinking. Reversing type 1 decisions has the potential of creating a lot of rework which is wasteful. This is why the right timing of type 1 decisions is critical.

Delay Architecture Decisions

Impacting decisions, mostly type 1, should be delayed until the probability that they would be questioned down the line is low enough. The right timing of type 1 decisions is a doubled-edged sword. Made too late, the resulting ambiguity is likely to significantly slow down design activities. Made too early, the rework needed to reverse them is likely to add significant delay and wasteful rework.

Lean product and process development promotes Set-Based Concurrent Engineering (SBCE) which helps to optimize when architecture decisions are to be made. Unlike traditional methods which lock design decisions too early or Agile methods which leave design decisions open too long, SBCE allows the final architecture design to emerge from teams' learning [Ward 2014].

Conventional development works in the following way:

Architecture and design decision incompatibilities are often identified too late in the process, which is a major cause of rework. The solution space is constrained too early in the process which can result in suboptimal architecture decisions.

In contrast, SBCE works like this:

"The set of design alternatives shrinks because they are eliminated, and as trade-off curves are developed, the remaining alternatives are developed to increasing levels of fidelity. Simultaneously, target ranges narrow in corresponding stages, converging on values that provide the best set of trade-offs."

Add Evolvability as a Key Non-functional Requirement

The authors of Building Evolutionary Architectures [Ford 2017] describe an evolutionary architecture as supporting "guided, incremental change across multiple dimensions". The key idea is to enable the incremental development of the product or system while preserving functional requirements such as scalability, elasticity, or resilience. As the product or system evolves during Agile iterations, its architecture qualities should not degrade over time.

The methods used to improve evolvability depend on the type of architecture. For example, loose-coupling and separation of concerns help architect software systems to evolve more gracefully. Let us illustrate this with clean architectures. Clean architectures such as the hexagonal one isolate the domain logic (the core) from non-core domain concerns such as inputs and outputs or persistence mechanisms. The code that implements the domain logic is protected from changes that could impact, for example, persistence mechanisms. The initial version of a piece of software could use an RDBMS and be ported later to a NoSQL DBMS with minimum rework.

Transforming type 1 decisions into type 2 decisions by making them easier to reverse also contributes to evolvability.

Sacrificial Architecture

Martin Fowler coined the term "sacrificial architecture" [Fowler 2014] to designate situations where a team deliberately chooses to throw away a codebase. Martin Fowler lists a few examples of companies who have done it, such as eBay® or Google®.

When the goal is to get rapid market feedback experimenting with an MVP, a sacrificial architecture is an option to consider as it would not be worth spending too much time designing an architecture that would have to change should the product owner decide to pivot.

Figure 13, “MVA – Forces and Heuristics” shows the relationships that link heuristics as well as the forces that influence architecture decisions and their timing.

Type 1 decisions can be delayed until the last possible moment, transformed into type 2 decisions by making the architecture more evolvable, or avoided with the creation of a sacrificial architecture.

Many forces influence the structure and timing of the architecture decision space:

When architecture decisions are made, it is important to document the motivations behind them. A minimum architecture documentation should be composed of a collection of Architecture Decision Records (ADRs) [Nygard 2011]. Each ADR describes a set of forces and a single decision in response to those forces. A simple command line tool such as https://github.com/npryce/adr-tools provides a lightweight way of documenting an architecture.

Michael Hammer, who is the author of a seminal HBR article [Hammer 1990], started the Business Process Re-engineering (BPR) movement in 1990. The key messages are:

The BPR movement enjoyed rapid growth for a few years before it fell out of favor following a number of large-scale BPR failures.

Hammer and Champy define process as: "a collection of activities that takes one or more kinds of input and creates an output that is of value to the customer" [Hammer 1993].

About at the same time, James Harrington [Harrington 1991] created a Business Process Improvement (BPI) method that defined the foundations of modern business process management practices.

James Harrington defines a process as: "any activity or group of activities that takes an input, adds value to it, and provides an output to an internal or external customer …​ There is no product and/or service without a process. Likewise, there is no process without a product or a service."

The three major objectives of BPI are:

A new role is defined – "process owner" – who is accountable for how well the process performs.

The method defines key process improvement concepts that are very similar to equivalent Lean concepts. Because the text was written at about the same time as The Machine that Changed the World [Womack 2007], we formulate the assumption that the authors (Harrington and Womack) either shared the same sources and/or invented the same concepts at the same time.

James Harrington recommends classifying activities into two categories:

NVA are often activities that exist because the process is inadequately designed, or the process is not functioning as designed.

BPI aims at reducing the process cycle time. It proposes a set of heuristics to compress cycle time. The equivalent Lean concept is the reduction of lead time.

BPI recognizes the need for big picture improvement when making incremental process improvement does not bring the desired result. The big picture technique requires stepping out of today’s processes and defines what perfect processes would be without the constraints of the present organization, processes, and/or technologies.

Process architecture is especially relevant to help understand when and how big picture improvement is required because it helps connect the operating model level to the individual process level. That is why it is useful in the context of Digital Transformation because operating models are likely to change.

Last but not the least, the author explains why feedback systems are very important. He recommends relating feedback to the individual performing the tasks, so they quickly understand their impact on quality and giving them the responsibility to take immediate action. This last point is likely to have been influenced by the Toyota® practice of the andon cord.

The Lean Enterprise Institute (LEI) defines value stream as: "All of the actions, both value-creating and non-value-creating, required to bring a product from concept to launch (also known as the development value stream) and from order to delivery (also known as the operational value stream)."

The Lean value stream definition is similar to James Harrington’s process definition. So why has Lean settled for a different terminology? James Womack, a co-founder of LEI, explains that the word "process" has different meanings (i.e., confusing process and procedure), therefore using value stream instead of process helps to remove ambiguity.

James Harrington defines the process owner role, while Lean defines the value stream manager or leader role.

The Lean definition brings an important distinction: development value streams to develop products and processes versus operational value streams.

Two main sources of complexity are product and service variety, plus the multiplication of touchpoints.

Product and Service Variety

Zeynep Ton observes: "Higher product variety and more promotions, in particular, increase costs all throughout the supply chain …​ More product variety and promotions also increase the likelihood of errors and operational problems in the stores." [Ton 2014].

Michael George recommends eliminating complexity the customer will not pay for and exploiting the complexity customers will pay for. He also recommends minimizing the costs of complexity offered [George 2004]. His approach includes analyzing core processes, identifying product families, and creating complexity value stream maps.

The author lists several levers to help manage complexity:

Multiplication of Touchpoints

In their book Beyond Advertising, the authors recommend thinking about brand ecosystems defined as "the brand’s multiple touchpoints and how they interact with each other, from a digital out-of-home experience to a tablet, from mobile to the store" [Wind 2016].

The authors observe that all interactions with a brand, from the first time you become aware that it exists to every touchpoint that you encounter along the way in your daily life, have an impact: "From the customer perspective, touchpoints with a brand or product are not differentiated: it is the seamless experience that matters."

Figure 14, “Touchpoints” illustrates the variety of touchpoints that a brand has to orchestrate to deliver a positive customer experience.

The authors predict that in the future: "Touchpoints will continue to multiply as we enter an era where every object has the potential to become connected and interactive."

This evolution impacts the enterprise as a whole: "New structures and processes will need to allow for agility and reaping the benefits both from decentralization and, when needed, the power that leveraging through centralization facilitates."

New operating models are emerging to support the ability to create real-time, personalized experiences.

The exploding number of combinations of product/service-to-touchpoint poses significant challenges to basic process design and operating model architecture.

Bain positions an enterprise’s operating model as a bridge connecting strategy and execution [Bain 2014]. Operating model design starts from a clear formulation of the enterprise’s value proposition. According to the Operating Model Canvas [Lancelott 2017], an operating model has six components:

An adaptive operating model is characterized by:

The accountability framework balances the autonomy agility required with effective alignment mechanisms.

The modular nature of the adaptive operating model allows for a "plug-and-pay" reconfiguration in response to evolving customer feedback.

Providing a seamless experience at every touchpoint requires linking customer experience to the underlying operational processes.

The starting point is the discovery of what creates value across a given journey from the customer’s point of view. By analyzing customer journeys, enterprises can pinpoint the operational improvements that will have the biggest effect on customer experience [Chheda 2017]. Once the desired operational improvements have been identified, the enterprise can implement them by activating four levers:

This document includes a revised service blueprint modeling technique to analyze customer journeys end-to-end. Figure 15, “Revised Service Blueprint Template” introduces a template that can be used to analyze customer journeys end-to-end.

For each stage of the journey, the diagram specifies the channel that supports the interaction with the user. A user story describes the set of interactions that occurs at that touchpoint.

For each of the user stories an analysis of required capabilities is performed. It starts with a functional description and can go as far as specifying the APIs that encapsulate corresponding business services.

When required capabilities are missing, an analysis of existing applications helps identify missing or incomplete ones. This gap analysis feeds an Agile requirement backlog. Backlog prioritization may result in customer journey changes.

The candidate services that would implement the required capabilities should be architected in a modular manner. This would facilitate the creation of new composite services. Reusable services can be aggregated into digital platforms that facilitate reuse and accelerate the customer journey’s reconfiguration.

The goal of an event storming workshop is to explore a domain with several people collaboratively.

To do so, people will be placing domain events on sticky notes on a wall along a timeline with an unlimited modeling surface.

The workshop puts together three kind of people: people with questions, people with answers, and a facilitator.

The orange color sticky note is the convention for the event.

The event is named with a past participle because of its simple semantic and notation. For example, order paid, product sent, etc.

Events are used because they are easy to grasp and relevant for domain experts.

Event storming workshops focus on domain events structured along a timeline.

The goal of the event storming workshop is to maximize the learning of all the participants. At the end of the workshop, the participants should have a shared knowledge of the domain subject of the workshop.

The physical set up is important: the surface of the wall represents an unlimited modeling surface and everyone should stand up to increase people’s engagement in the workshop. There must be a continuous effort of every participant to maximize the engagement. The collaborative approach involves people with questions, people with answers, and a facilitator.

Markers and sticky notes should be available in different colors and in sufficient quantity.

There should be no limitation of the scope under investigation and the model should be continuously refined through low-fidelity incremental notation.

Event storming can have different goals, but mainly we can distinguish these two commons ones:

The concepts used during an event storming session, apart from the event concept, is voluntarily set by the participants whenever they feel they need one. However, the facilitator can suggest the following "usual" concepts:

First, before diving into the event-driven architecture style, we will define the Command, Query, and Event concepts and their relations regarding time and state management:

Figure 17, “Concepts of Command/Query/Event and their Relation to Time” illustrates these concepts related to time.

Command

A command represents the intention of a system’s user regarding what the system will do to change its state.

Query

A query asks a system for its current state as data with a specific model.

Event

An event represents a fact about the domain from the past; particularly, on every state change performed by the system, it will publish an event denoting that state mutation.

Events characteristics:

Events are published whenever there is an interaction with the system through a command (triggering or not a state transition; if not, the failure is also an event) or a query (no state transition, but the interaction is interesting in itself, such as for analytics purposes).

Better Handling of State, Concurrency, and Time

Basically, command and query represents the intention of the end users regarding the system:

This distinction relates to state and time management as well as expressing what the user wants the system to do and, once the state has mutated, the system will publish an event notifying the outside world that something has happened.

The Real World is Event-driven!

The world is event-driven as the present is very difficult to grasp and we can only clearly separate past and future. Past is the only thing we can – almost – be sure of and the event way of describing the result that occurred in the past or should occur with the future system is to use an event. It is as simple as "this happened". The event storming workshop format is one of the most efficient and popular ways of grasping a domain for people involved in software design. (See Chapter 11, Event Storming Workshop for a description of the technique.)

Loose-coupling & Strong Autonomy

Event mechanisms loosen the coupling between the event publisher and the subscribers: the publisher doesn’t know its subscribers and their numbers. This focus on event also enforces a better division of concerns and responsibility as too many events or coarse-grained events can be a design issue.

Focus on Behavior and Changeability

Commands and events force software designers to think about the system’s behavior instead of too much focus on its structure.

Software designers and developers should focus on separating their domain logic between:

This separation of different concerns leads to greater changeability of software systems.

Better Operability with Events: Stability, Scalability, Resilience, and Observability

Commands can be used to better represent the user’s intention and fit well with deterministic domain logic that benefits from consensus algorithms such as the Raft Consensus Algorithm (see https://raft.github.io/) to distribute the execution of business logic on several machines for better resiliency and scalability along with some sharding logic (see http://www.startuplessonslearned.com/2009/01/sharding-for-startups.html).

Good operability needs good observability of the running systems and this is reached by strong logging practices. Good logs are actually events in disguise; replaying a system behavior through logs is actually following the flow of technical and domain events that exhibit what the system has done.

The event store becomes the source of truth and the system’s end state is the end result of applying all theses events. Event-driven architecture doesn’t mean event sourcing.

The main motivation of CQRS to use these dedicated models is to simplify and gain better performance for interactions with the system that is unbalanced (read-intensive/write-scarce or write-intensive/read-scarce interactions). If CQRS simplifies each model in itself, the synchronization and keeping all the models up-to-date also brings some complexity. CQRS can be implemented sometimes without proven requirements and can lead to some over-engineering.

Every command, query, and event should share the same metadata, essentially telling us who is emitting when, where, and what is the relevance, and also each artifact should be uniquely identified. This metadata can be the following:

The Clound Native Computing Foundation issued a specification describing event data in a common way (see https://cloudevents.io/).

Events originate from bounded context that must be explicitly defined and then they can be consumed by others' bounded contexts. The traditional DDD’s context map and the various strategic patterns of bounded context integration are very useful tools for mapping the landscape.

We also advocate to translate each event that comes from another bounded context to a command of the context that consumes it to denote explicitly the intention behind the event’s consumption.

To ensure global consistency across multiple systems we need mechanisms to "rollback" or compensate the effect of applying a command to get back the systems’s global state to a consistent one (a consistent state does not means to the state that existed before applying all those commands; a consistent state is where all the sub-systems are consistent with their data).

As an example, think about the way your bank "cancels" a contentious movement on your account; the bank doesn’t remove the contentious movement, instead it issues a new movement compensating the effect of the bad one. For instance, given a contentious debit movement of $100, the bank issues a credit movement of $100 to get back to a consistent balance even if the movements list of the account now exhibits two movements cancelling each other.

As a first step, we need to identify the inverse of each command that will cancel the effect or "compensate" a former one. The Saga patterns describe the structure and behavior to attain such a consistency goal: in case of failure of one command, the other services issue new commands that compensate the former one, hence "rollback" the whole distributed transaction as a result. Two types of the Saga pattern exist: orchestration and choreography.

Command, Query, and Event as the Interface of the Software System

Command/Query/Event Declaration

Command, Query, and Events are represented using a data record structure; they should not have any operations associated with them, contrary to a value type, for instance. Each one should have an identifier, even if they are immutable, to reference them easily.

Command/Query/Events are respectively declared with the "defcommand", "defquery", and "defevent" macros. The first argument is a keyword for naming the concept, and then a keyword of a spec describing the payload of that command, query, or event. For example:

Hexagonal Architecture

Hexagonal architecture decouples the application, domain, and infrastructure logic of the considered system. It does so by using interfaces (ports) located in the domain part and implementations (adapters) located in the application and infrastructure parts that are wired into the domain. The event bus abstraction belongs to the infrastructure part of the architecture; several implementations can then fulfill the infrastructure requirements (distributed or not), forwarding events to the browser with Server-Side-Events (SSE).

Hexagonal architecture allows effective decoupling of application, domain, and infrastructure concerns in a back-end system. In our opinion, hexagonal architecture is closely related to event-driven architecture as it allows an effective decoupling of the event publication and domain logic with the implementation details of the event infrastructure that can be quite tedious.

Bus Abstraction

The event bus has two sides: publishing events and subscribing to events. These two operations are exposed on an interface that can be implemented with various means, hence being a Service Provider Interface in a hexagonal architecture.

Therefore, bus implements the following protocol:

Each event needs to be classified depending on a value it holds; the exact value extracted from the event needs to be parameterized depending on the bus, but each bus should do it in a homogeneous way. So, the bus creation needs to have a key extraction function as an argument; usually it will be whether the organization that is concerned by this event – the ":org-ref" – or the event type ":event-type" to which every event belongs.

Trigger Reaction to Events for Subsequent Domain Logics

Some events can trigger other domain logic (e.g., a search query triggers adding this query to the user search history). Events here allow the decoupling of the business logic between the publishing context and the reacting context.

Event Publication

Domain logic that sits in back ends publishes events on every API interaction from the clients. These events are published through the EventBus interface with various implementations detailed in the following sections.

Kafka Messaging System

Kafka is used as the distributed messaging system that collects and distributes events between sub-systems.

Kafka Subscription

Back-end Bus Systems

Multicast Event Bus

Kakfa Event Bus

In-memory Event Bus

Metrics Exposition with Prometheus through Event Subscription

Metrics are a numeric representation of data measured over intervals of time. Metrics can derive knowledge of the behavior of a system over intervals of time in the present and future. Events counting through Prometheus counters (see https://prometheus.io/) is a great way to observe the system’s behavior over time. Metrics have:

Back ends expose technical and domain metrics through Prometheus endpoints for monitoring and alerting concerns. The business events are a strong proxy to any technical incidents; monitoring domain events should be the main values to monitor over time.

The biggest advantage of metrics-based monitoring over logs is that, unlike log generation and storage, metrics transfer and storage has a constant overhead. Metrics are also better suited to trigger alerts over aggregation of events.

Events Push to Connected Clients/Users with SSE Subscription

Once a user connects to the application, the browser app needs to collect all the events related to the organization to which this connected user belongs.

At different levels of an organization, stakeholders and customers need to align on what is being delivered, and ascertain that goals are being achieved and policies adhered to. This alignment is done among continuous change driven by demand both internal and external – competitors, regulation, innovation, operations. A certain level of control and management is required.

The TOGAF standard defines governance as: “the ability to engage the involvement and support of all parties with an interest in or responsibility to the endeavor with the objective of ensuring that the corporate interests are served, and the objectives achieved” [TOGAF 2018].

[COBIT 5]^[1], in Principle 5, makes an important distinction between the management activities/processes and governance itself.

Governance ensures that enterprise objectives are achieved by evaluating stakeholder needs, conditions, and options; setting direction through prioritization and decision-making; and monitoring performance, compliance, and progress against agreed direction and objectives.

Management plans, builds, runs, and monitors activities in alignment with the direction set by the governance body to achieve the enterprise objectives.

Another standard, [ISO/IEC 38500], applies to the governance of management processes (and decisions) relating to the information and communication services used by an organization. The standard provides a structure for governance of IT to assist those at the highest level of organizations to understand and fulfill their legal, regulatory, and ethical obligations regarding their organization’s use of IT.

ISO/IEC 38500 is driven from the top down; IT departments need to make sure that they are ready for the new demands the board will pose (e.g., performance measurements, clear governance mechanisms). One criticism of both COBIT and ISO/IEC 38500 is that these frameworks are reactive and don’t adapt to quick change well; an example is change outside budgetary cycles, and the time it takes to enact these changes.

Agile doesn’t contradict both the COBIT and ISO/IEC 38500 definitions and guidance; in fact, it emphasizes the interaction of individuals, collaboration, and ensures the stakeholder objectives/goals are achieved. Often in non-Agile organizations, ownership is fluid between business and IT, whereas in Agile organizations the business stakeholder (product owner) "owns" the value chain. This risk has been reduced in Agile IT governance frameworks as it focuses on recognizing the right person as the suitable owner of the portfolio/program/project at the right level from inception to delivery.

"To efficiently create chaos, Langdon realized, requires some order." (Dan Brown)

There is a need for procedures and policies to be in place to reconcile sometimes what can be described as conflicts of interest, agreements, responsibilities, and rewards, and update stakeholders on "contracts" that have been made.

Critics of Agile say that Agile methods allow teams to work in an unstructured way, which prevents clear lines of accountability and discourages documentation. On the other hand, proponents of Agile delivery argue that the methods rely on information on the current status of the project being visible to the whole business, instead of central processes for command and control. They also believe that, because the methods are designed to be self-assuring, there is proper governance and accountability built into Agile practices. Control processes, in Agile, are more collaborative and are run continuously by the business owner of the product or service. On the other hand, critiques of the status quo argue that “IT governance is killing innovation” [HBR 2013]; governance does not allow us to respond quickly enough to new demand.

Both camps agree that communicating on the status, progress, and budget and satisfying regulatory and auditing requirements are required.

Governance in Agile is a balancing act; on one hand autonomy, and on the other accountability to a “contract”, freedom to innovate versus following proven routines and policies [HBR 2017].

On this same point, Isdell’s “freedom within a framework” [Kesler 2008], governance became the means to corralling an accepted level of chaos – a way to engage the natural tension between many new global initiatives and the need for geographic General Managers to get more aggressive about finding local solutions to brand, product, and revenue gaps.

Increasingly businesses, in response to competitive pressures and new Agile competitors, are adopting a product-led approach to their enterprises which prompts their customers to use their products and share them with their networks. These products and networks run on "platforms" that have a significant viral effect (network effect: https://en.wikipedia.org/wiki/Network_effect).

What has this to do with Agile governance? Agile teams, and in consequence Enterprise Architecture, co-create and deliver customer-centric products (not projects). The implication is the need for cross-functional teams (business and technology) that are product-led. This poses a huge challenge to "legacy" data in siloed organizations that need to reinvent their own organizations and operating models. Other aspects to consider are metrics, performance, and incentives; how to recognize success or failure?

An Agile governance structure needs to recognize these new ways of working and to measure value.

The following sections assume you are on a journey to a customer-centric product-led organization where "Agile" is the way of working [McKinsey 2016].

"Product-led" Agile Enterprise Architecture Governance

"At PepsiCo®, we are leveraging design to create meaningful and relevant brand experiences for our customers any time they interact with our portfolio of products." (https://hbr.org/2015/08/pepsicos-chief-design-officer-on-creating-an-organization-where-design-can-thrive)

A product or service is delivered to a customer via a series of activities; the chain of activities that delivers a valuable product or service to a customer is usually referred as a "value chain". The concept was first described by Michael Porter [Porter 2004].

An Agile Architecture governance structure could look like the following diagram.

Key Areas

Within an enterprise, governance structures exist that need to adhere to internal policies and external regulatory frameworks. Corporate and IT governance are such "bodies" that with support of frameworks (like COBIT and ISO/IEC 38500) help enterprises to manage such adherence. In a product-led organization, customer focus is essential and has increased regulatory attention [PWC 2013]. Architecture is represented by the organization’s Chief Architect or delegates on each of the governance boards as required.

The Enterprise Technology Board is a decision and policy-making group charged by the CIO/CTO with driving and influencing the Enterprise Architecture of the organization aligned with its strategic goals and objectives. Architecture should not impose a model nor define it; it should be an outcome as a result of collaboration within the enterprise – IT and business. As part of the Centers of Enablement and in conjunction with the business and product owners, architecture is defined and enabled via Innovation Hackathons. These innovation hackathons play a major part in enabling technology innovation and collaboration within the enterprise but also in partnership with external stakeholders such as vendors. Co-creation is an important feature of these enablement centers.

Within the product value chain, the product owner, with the support of the product architect, prioritizes features considering customer needs and existing technology roadmaps ("near-term roadmapping"). The impact of changes as a result of constant refinement is governed within the Product Architecture Board.

Teams make architectural design decisions; de-centralized decision-making focused on architecture and design quality and ensuring adherence to enterprise principles – but also these decisions are future-proof (business and technology vitality) and adhere to the solution Definition of Done (DoD) (testable, scalable, accepted by the product manager, regulatory/compliance standards met).

New digital applications or platforms need to integrate with legacy systems. For example, new digital banking applications are likely to need the services provided by legacy core systems or Enterprise Resource Planning (ERP).

Effective integration mechanisms are required because pure greenfield or "big bang" approaches are not realistic. However, such integration mechanisms are not easy to architect, implement, and operate. The main challenges are:

The ambition of this playbook is to provide a roadmap to help progressively architect the overall system into a loosely-coupled one. Figure 23, “Monolithic to Modular Journey” illustrates this journey.

The steps toward modularization are described below:

In order to facilitate steps 1 and 2, instead of a relying on a single API with a flat set of endpoints, Etsy® (see https://www.etsy.com/developers/documentation/getting_started/api_basics) created a two-layer API using meta-endpoints. Similar to a pattern used by Netflix® and eBay’s ql.io, each of Etsy’s meta-endpoints aggregates several other endpoints. This enables server-side composition of low-level, general-purpose resources into device or view-specific resources.

Since the heyday of SOA the idea of creating an API layer above legacy systems is seen by many as the magic bullet that can solve a majority of legacy integration problems.

An API is an application software intermediary that enables a software program to interact with other software. In the context of trading, an API can, for example, enable your software to connect with a broker to obtain real-time pricing data or place trades.

The quality of an API is constrained by the quality of the software that publishes it. A good API shields the internal complexity of the software that provides it and presents data in a usable way. Because legacy software is often complex and monolithic, it is a challenge to design APIs that are not prone to abstraction leak. Abstraction leak refers to a situation where an API consumer has to understand unnecessary implementation details to use it. When it happens it creates unintended coupling that puts the system’s agility in jeopardy.

Designing APIs requires business domain knowledge. Technologies such as API managers provide little if any help solving the hard problems which are:

Designing APIs that external developers will love is key to creating the kind of digital ecosystems that characterize many digital business models. Designing the information architecture that will guide API definition does not happen in a vacuum. Best practices show that it is driven by business goals, persona modeling, and task analysis.

Legacy applications when they are systems of record hold the authoritative version of data. For example, a payment application that makes a credit transfer. Primary data is generated by a system of record when it processes a business event; it cannot be derived from other data.

When integrating a new application with a legacy one that manages primary data, it is important to distinguish when:

In the latter case, it is safer to let the legacy application modify the primary data because it implements the business rules that ensure the modification will be performed in the right way. The new application could become the new system of record if it would implement equivalent business rules. This is often impractical because legacy applications tend to be poorly documented.

When a legacy application exposes a "write" API, it may increase the transactional load to a level that is not sustainable. For example, the third-party securities lending system of a global custodian could not include custody system calls within the boundary of a distributed transaction. In addition, developing APIs on top of some old technologies such as IMS/DC may present specific technical challenges.

Using the synchronous API style, in particular when write-APIs are in scope, decreases the ability of the system to stay responsive in the face of a failure (lower resilience). It also decreases its ability to respond in a timely manner because it has to wait for the response of a legacy application’s API (less responsive).

When the synchronous API style has drawbacks that exceed its benefits, the alternative is to use asynchronous message passing which leads to a new kind of architecture design that is characterized by:

Asynchronous message-passing still supports business transactions, but does it using Sagas. The Saga pattern describes how to implement business transactions without two-phase commit as this does not scale well, in particular in cloud-native systems.

The business transaction is divided into multiple steps or activities. The Saga pattern has the responsibility to either get the overall business transaction completed or to leave the system in a known termination state. So, in case of errors a business rollback procedure is applied which occurs by calling compensation steps or activities in reverse order. This pattern is not new, though it was not named Saga. For example, the idea of compensating transactions has been used in the past to process payments or securities handling business transactions.

Unlike two-phase commit transactions that are handled automatically by the database or the middleware, the Saga pattern may require specification of specific business logic to handle the consequences of business events. For example, if the securities that have been loaned (third-party lending) are sold, the securities lending system will replace them with other available securities. If the securities are no longer available, the system has to inform the trader and assist her resolving the issue.

Business objects that have an identity which remains the same throughout their life are modelled as "entity objects". Clients, trades, and securities are examples of entity objects which experience state changes during their life. It is important to identify entity objects in a way that spans the life of a system and can extend beyond it. This is key to successful integration with legacy systems because entity objects are likely to be replicated.

The example of When Issued (WI) transactions illustrates that identity management is more than specifying tables’ primary keys. For example, a treasury bond can be purchased or sold when it has been authorized but not yet issued. A dummy CUSIP number is created to identify the security before CUSIP Global Services (CGS) creates the official one. The WI trade is conditional and settlement can occur only when the security has been listed and has been attributed with an official CUSIP number.

Some changes of an entity object state can have business consequences. For example, in the case that the WI security is not listed or admitted to trading, all transactions effected during that period are declared void by the exchange. This state change is a business event that triggers downstream actions to undo the WI trades. That is why asynchronous communication using IBM MQSeries® is not sufficient. Communicating events (state changes) is better suited to express dynamic business rules. Just sending messages that represent the state of an entity at a point in time does not inform the listener that a business event happened.

An entity object has attributes that describe its state. A state change of an entity object modifies the value of some of its attributes. Depending on the context, some attributes may not be relevant. For example, the front office does not need some of the data that will be required at settlement time.

Master data is about managing in a consistent manner the identifiers and key attributes of core entity objects of the bank such as Party, Security, or Account. Referential data can be fully managed by new digital applications when they implement all the business rules required to modify primary data. When this is not the case, derived master data can be replicated in new digital systems and accessed in read-only mode: updates being still handled by "legacy" systems of record.

Too many legacy systems are polluted by inflated master data structures that aggregate all the data that describe entities regardless of their context and usage. This tends to increase the overall system’s complexity and promote high coupling. When integrating new digital applications with legacy ones, it is preferable to protect new software code from unnecessary complexity that can pollute it.

The illusion of the "all encompassing" enterprise data model needs to be replaced by a modular way of architecting models. Modeling should be driven by use-cases specific to each context. For example, trading is different from clearing and settlement though some entities will span more than one context. When interfacing a new digital cloud-native application with a legacy one, an anti-corruption layer can be created to shield the new code from legacy complexity. The anti-corruption layer can mediate or translate old data structures into cleaner ones that meet digital application needs.

When designing the resource model of a RESTful API or an event data structure, business domain concepts should prevail over constraints imposed by legacy data structures. Finding the right balance between clean versus unclean design cannot be solved by technology alone; it requires business domain expertise.

The simplified diagram in Figure 24, “Legacy Integration Strategies” represents how to combine integration patterns.

APIs can open legacy applications’ functionality to the outside.

An entity can be replicated in more than one system as long as its identity is preserved and data lineage properly managed. REST resources map to business entities. Derived data is formatted to meet the needs of the data consumer and cached to minimize network traffic.

Legacy applications publish events that model entity state changes that are of interest to digital cloud-native applications. Sagas whose scope spans digital and legacy help maintain the overall system’s (eventual) consistency.

The diagram also shows that the legacy application manages Entity D, which could be Instrument. If the Instrument’s price changes, the legacy application can publish a business event that represents that change. A new digital application that consumes the price change event can, for example, trigger a state change of limits to the order it manages. The digital application emits a new event that ascertains that the order (Entity E3) has been released for execution. This type of causality chain can be expressed in a graph that can drive Saga logic.

We believe that architecture models that only represent a system statically are incomplete. They need to be completed by models that represent the system’s behavior. The scalability, resilience, and responsiveness of a system cannot be verified in the absence of dynamic modeling.

Monolithic legacy systems often combine what would be several bounded contexts in a well-modularized system. The approach consists of identifying these candidate bounded contexts and starting to develop new application components at the edge of the legacy system. For more details on the strangler pattern, see Chapter 15, Strangler Pattern.

Figure 25, “Strangling the Legacy Monolith” illustrates the process.

A new application component bc1 (see https://www.etsy.com/developers/documentation/getting_started/api_basics) is developed using a cloud-native application style (e.g., microservices). The application component bc1 is interfaced with the legacy system using patterns described in Figure 24, “Legacy Integration Strategies” (in particular the anti-corruption pattern). The corresponding functions and features are removed from the legacy system.

The process repeats itself to develop other bounded contexts (bc2 to bci) at the edge of the legacy systems. When the last bounded context (bcn) is developed, the legacy system can be decommissioned. It has been strangled!

Some recommend a more aggressive approach; they claim that you should starve your monolith to death (see https://read.acloud.guru/if-you-cant-strangle-the-monolith-starve-it-to-death-fcc824d3c82). Because decommissioning legacy systems requires significant investment, attention should be given to the economic side of the equation. It is good to starve your monolith rapidly if the business case proves positive.

Conclusion

The approach we have described is key to better integrate digital applications with legacy ones and to ultimately decommission them. However, the reader may think that the learning curve is too high and the enterprise is not up to the challenge.

The obvious alternative would be to develop new digital capabilities with the same old architecture models mastered by the IT organization. This is not such a good idea because:

The problem space holds what the enterprise does – its business capabilities – to keep it running and able to operate. A business capability is a specific function or ability the enterprise possesses in order to achieve its goals.

The problem space describes several things:

The problem space holds the domain within which the enterprise operates and represents the world as we perceive it; it describes the Business Architecture.

Sub-domains

A domain can be decomposed into sub-domains which typically reflect some organizational structure. Sub-domain boundaries are determined in part by communication structures within an organization. The sub-domains are stable; they change only for strategic reasons and are independent of software.

Example of an E-commerce System

An e-commerce system consists of a product catalog, an inventory system, a purchasing system, and an accounting system, etc. They are sub-systems in that the system as a whole is partitioned into them. The system is partitioned in this specific way because the resulting sub-systems form cohesive units of functionality.

How to Identify Sub-domains

Domain knowledge is key to decomposing a domain into sub-domains that have a high level of internal cohesion and minimum dependencies with other sub-domains. Conducting event storming workshops is a great way to accelerate the acquisition of domain knowledge and explore domain decomposition scenario. The event storming workshop technique is introduced in Chapter 11, Event Storming Workshop.

Distillation

The enterprise operates with several sub-domains. Depending on its business, some are generic (such as accounting or HR), some are support, and some are core, meaning the current strategy directly relies on the core domains to attains its goal. Not all parts of a large system will be well designed.

Strategy is defined as: "Strategy describes the organization;s objectives according to the environment and the available resources, then the resources allocation in order to create value for the clients along with profits for the organization and its employees."

The business capabilities are almost the same for different enterprises involved in the same business, but their implementations – the solution space – will differ. While sub-domains delimit the applicability of domains, bounded contexts delimit the applicability of domain models. As such, the bounded context is within the solution space.

Bounded context is the solution as we design it. It describes the software architecture and is used to manage the complexity, and is therefore linked to the business.

Bounded context means different models of the same thing (e.g., book, customer, etc). Bounded context is represented by models and software that implement those models. This is where we find patterns and heuristics.

Domain Model and Ubiquitous Language

The ubiquitous language is a deliberate language designed to be unambiguous and on which all stakeholders agreed. This language is found in every artifact manipulated by the stakeholders (UI, database, source code, documents, etc.). The concepts conveyed by the domain model are the primary means of communication; these words should be used in speech and every written artifact. If an idea cannot be expressed using this set of concepts, the designers should iterate once again and extend the model, and they should look for and remove ambiguities and inconsistencies. The domain model is the backbone of the ubiquitous language.

Bounded Context

A bounded context delimits the applicability of a particular model so that team members have a clear and shared understanding of what has to be consistent and how it relates to other contexts. Bounded contexts are not modules,

Bounded contexts separate concerns and decrease complexity. A bounded context is the boundary for the meaning of a model. A bounded context creates autonomy, hence allowing a dedicated team for each. Bounded contexts simplify the architecture by separating concerns.

How to Identify a Bounded Context?

Conflicts of naming suggest different contexts.

Context Map

A context map describes the flow of models between contexts and provides an overview of the systems landscape. A context map help to identify governance issues between applications and teams. It helps us to see how teams communicate and their "power" relationships. With a context map we get a clear view on where and how bad models propagate through IS landscapes.

You can use the metaphor of a river flowing to describe the relations between two bounded contexts: if you are upstream and pollute the river, the downstream people will be impacted - not the opposite.

A relationship between two bounded contexts in which the upstream group’s actions affect the downstream group, but the actions of the downstream do not affect the upstream. It is not about the data flow’s direction, but about the models' flow.

We can categorize the context map patterns in three categories:

Upstream Patterns

Open Host Service

Event Publisher

Domain events are something that happens in the domain and that is important to domain experts. An upstream context publishes all is domain events through a messaging system (preferably an asynchronous one) and downstream contexts can subscribe to the events that are relevant for them and conform or transform those events in their models (following an ACL) and react accordingly.

Midway Patterns

Shared Kernel

Published Language

Separate Ways

Partnership

Downstream Patterns

Customer/Supplier

Conformist

Anti-corruption Layer

Mapping the Context Map Patterns

We can organize the context map patterns along two axis: Control and Communication.