Digital Practitioner Body of Knowledge™ Standard

We saw in Governance, Risk, Security, and Compliance how governance emerges, as a response to scale and growth, and the concerns for risk and assurance in the face of increasing pressures of the external environment. One important response has been the emergence of the line _versus staff_ distinction. As Christian Millotat (and many others) have noted: "[m]any elements that have become integral parts of managerial economics and organizing sciences can be traced back” to military staff systems [197 p. 7]. These include:

Staff functions in the enterprise include planning, coordination, and operations; broadly speaking, and with key differences depending on the industry, the following are considered “staff":

While the following are considered “lines” (analogous to the warfighting units in the military):

Enterprise Architecture has as a key part of its mission the task of collecting and combining knowledge to support decision-making. Therefore, an Enterprise Architecture organization can be seen as a form of staff organization. Most often it is seen as a specialized staff function within the larger staff function of IT management, and with the increased role of digital technology, there are corresponding pressures to “move Enterprise Architecture out of IT” as we will discuss below.

The classic line/staff division is a powerful concept, pervasive throughout organizational theory. But it has important limitations:

In the history of line versus staff relations, we see tensions similar to those between Enterprise Architecture and advocates of Agile methods. The challenges, debates, and conflicts have only changed in their content, but not their essential form.

But in many cases, centralizing staff expertise and the definition of acceptable practices for a domain is still essential. See the discussion of matrix organizations and even feature versus component teams.

In terms of overall positioning, Enterprise Architecture is often portrayed as mediating between strategy and portfolio management (see EA Context, Based on Ross, derived from [237 p. 10], Figure 1-2).

Notice the distinction between Enterprise Architecture as a capability and as an artifact. The practice of Enterprise Architecture is not the same as the actual Enterprise Architecture. For the purposes of this document, we define Enterprise Architecture’s concerns as essentially the enterprise operating model: process, data, organizational capabilities, and systems.

One of the most frequently used visualizations of Enterprise Architecture’s concerns is the Zachman Framework (see A Variation on the Zachman Framework, loosely based on [313] and succeeding work).

We were exposed to the data modeling progression from conceptual to logical to the physical data model in Information Management. The Zachman Framework generalizes this progression to various views of importance to organizations, as shown in the columns:

Overall, the Zachman Framework represents the range of organizational operating model concerns well. Certainly, sustaining a large and complex organization requires attention to all its concerns. But what good does it do to simply document the contents of each cell? Such activity needs to have relevance for organizational planning and strategy; otherwise, it is just waste.

It is reasonable to associate the Business Model Canvas with strategy, and the Zachman Framework with the Enterprise Architecture as an artifact (see Business Model versus Operating Model).

This distinction helps us position the Enterprise Architecture group with respect to key partner organizations:

Organizational Strategy

The operating model needs to support the business model, and so, therefore, Enterprise Architecture needs a close and ongoing relationship with organizational strategists, whether they are themselves line or staff. Defining digital strategies is a challenging topic; we have touched on it in Digital Context, Product Management, and Investment and Portfolio. Further discussion at the enterprise level will be deferred to a future edition of this document.

Portfolio and Investment Management

Architecture needs to be tied to the organization’s investment management process. This may be easier said that done, given the silos that exist. As Scott Bernard notes: “Enterprise Architecture tends to be viewed as a hostile takeover by program managers and executives who have previously had a lot of independence in developing solutions for their own requirements” [29], Case Study Scene 1.

Many organizations have a long legacy of project-driven development, in which the operational consequences of the project were too often given short shrift. The resulting technical debt can be crippling. Now that there is more of a move towards “you build it, you run it” the operability aspects of systems are (perhaps) improving. However, ongoing scrutiny and management are still needed at the investment front end, if the enterprise is to manage important objectives like vendor leverage, minimizing technical debt, reducing investment redundancy, controlling the security perimeter, and keeping skills acquisition cost efficient (more on this below in the section on Enterprise Architecture value).

Portfolio management is discussed in depth in a subsequent Competency Category.

Infrastructure and Operations (I&O)

Finally, the Enterprise Architecture group often has a close relationship with I&O groups. This is because in organizations where operations is a shared service, the risks, and inefficiencies of technical fragmentation are often most apparent to the operations team. In organizations where operations is increasingly distributed to the application teams (“you build it, you run it”) the above may be less true.

Other staff organizations that may develop close relationships with Enterprise Architecture include vendor management and sourcing, risk management, compliance, and security. Notice that some of these have strong governance connections (although we do not consider “governance” itself to be an organization, which is why it was not included in the discussion above).

We have covered the concept of cost of delay previously, at the team and product level. However, this powerful concept can also be applied at higher levels. Cost of delay is not a concept familiar to most architects. It poses two important challenges:

As we have discussed previously, the definition of cost of delay is intuitive. It is the opportunity cost of not having a given product or service available for use: the foregone revenues, the cost of the workarounds and inefficiencies. If the architecture process becomes the critical path for a product or service’s release (a common experience), then the architecture process is responsible for that product’s or service’s cost of delay.

The cost of delay can take various forms, some of them significant. For example, suppose there is a need to demonstrate a product to key clients at a trade show. This could be the company’s best opportunity to develop business; the sales team estimates $12 million in funnel opportunities based on previous experience that should result in at least $2 million in sales in the year, with projections of another $1 million in maintenance and renewals. However, if the product is not ready, these benefits will not materialize. If everything else is ready, but the architecture process is delaying product readiness, then the architecture process is incurring $2 million in the cost of delay. This is bad. The architecture process is clearly impacting significant business objectives and revenue.

But the question still needs to be asked: “what benefits do we receive from having an architecture process?”. We discussed such benefits above. Are these benefits adding up to $2 million a year? No? Then your architecture process does not make good economic sense. On the other hand, what if the architects were kept out of the picture, and the product team chooses an untested technology, instead of re-using a well-known and reliable approach already proven for that company? What if that decision were the cause of missing the trade show? What if it can be shown that re-usable components identified as architecture standards were increasing the speed of delivery, and reducing the cost of delay, because they are reducing the need for product teams to perform risky (and yet redundant) R&D activities?

Quantifying these benefits across a portfolio is difficult, but should be attempted. Cost of delay can and should be calculated at the portfolio level, and this can provide “enterprise-level decision rules” that can help an organization understand the cost and value of operating model changes [229 pp. 35-38] including instituting processes (such as change management or technology lifecycle management), or even the establishment of Enterprise Architecture itself.

We touched on technical debt in Application Delivery's discussion of refactoring. Technical debt is a metaphor first introduced by Ward Cunningham [78] in the context of software, widely discussed in the industry. It can be applied more broadly at the portfolio level, and in that sense, is sometimes discussed in Enterprise Architecture. Debt exists in the form of obsolete products and technologies; redundant capabilities and systems; interfaces tightly coupled where they should be loose and open, and many other forms.

Technical debt, like the cost of delay, can and should be quantified. We will discuss approaches to that in the Competency Category on portfolio management.

We have often referred to the concept of common ground through this document. The architecture supports common ground understanding at scale, by curating a shared mental model for the organization. In doing so, it enables the “right emergent behaviors” (as Adrian Cockcroft suggests). It also enables communication across diversity and may improve staffing flexibility and mobility among teams.

Evidence of Notability

Architecture has been a metaphor for digital systems strategy and design since the 1960s. It has given rise to multiple professional organizations, and frameworks, and much literature.

Limitations

Architecture (all practices) is encountering resistance from digital-first organizations and its messaging and value proposition needs to evolve.

Related Topics

Dan Moody notes [199]:

Visual representations are effective because they tap into the capabilities of the powerful and highly parallel human visual system. We like receiving information in a visual form and can process it very efficiently: around a quarter of our brains are devoted to vision, more than all our other senses combined [63]. In addition, diagrams can convey information more concisely [27] and precisely than ordinary language [8, 68]. Information represented visually is also more likely to be remembered due to the picture superiority effect [38, 70] …​ Visual representations are also processed differently: according to dual channel theory [80], the human mind has separate systems for processing pictorial and verbal material. Visual representations are processed in parallel by the visual system, while textual representations are processed serially by the auditory system …​

As the above quote shows, there are clear neurological reasons for diagramming as a communication form. To expand a bit more on the points Dan Moody is making:

Human vision uses parallel processing. This means that a given image or visual stimulus is processed by many neurons simultaneously. This is how we can quickly recognize and act on threats, such as a crouching tiger.

A large percentage of our brain is devoted to visual processing. You will see figures quoted from 25% to 66% depending on whether they are “pure” visual tasks or vision-driven tasks involving other brain areas.

The old saying "a picture is worth a thousand words" is consistent with the science. Diagrams can be both faster and more precise at conveying information; however, this has limits.

Finally, pictures can be more memorable than words.

Architects and architecture are known for creating diagrams — abstract graphical representations of complex systems. The first known instance of applying graphical techniques to a digital problem was in 1947 (see The First Software Flowchart ^[1][294]), and visual notations have evolved along with the field of computing ever since. Notable examples include:

(We touched on data modeling in Information Management). We will examine the ArchiMate modeling language, a standard of The Open Group, for a current and widely-used notation, in more detail in a future Competency Category.

Research at Microsoft suggests that developers use diagrams for four purposes:

They argue “diagrams support communicating, capturing attention, and grounding conversations [4]. They reduce the cognitive burden of evaluating a design or considering new ideas [13]” [58].

But visual notations have been problematic in the Agile community; as Fowler notes [103]. There is no question that some IT professionals, including perhaps some of the most skilled software engineers, find little of use in diagrams. As Martin Fowler says: “people like Kent [Beck, eXtreme Programming originator] aren’t at all comfortable with diagrams. Indeed, I’ve never seen Kent voluntarily draw a software diagram in any fixed notation”. However, it seems likely that Kent Beck and others like him are members of a programming elite, with a well-honed mental ability to process source code in its “raw” form.

However, if we are building systems to be operated and maintained by humans, it would seem that we should support the cognitive and perceptual strengths of humans. Because diagrams are more readily processed, they are often used to represent high-level system interactions — how a given service, product, or application is related to peer systems and services. Building such depictions can be helpful to fostering a shared mental model of the overall system objectives and context. The more complex and highly-scaled the environment, the more likely such artifacts will be encountered as a means to creating the mental model.

The strength of human visual processing is why we will (probably) always use graphical representation to assist in the building of shared mental models. Specialists in the syntax and semantics of such designs will therefore likely continue to play a role in complex systems development and maintenance. Currently, if we seek to hire such a specialist, we recruit some kind of architect — that is, the professional role with the skills.

Note that flowcharts, data models, and other such diagrams tend to be associated more with the idea of “solutions” or “software” architecture. We will cover the architecture domains in the next Competency Category, including examples of Business Architecture diagrams.

Visualization has a number of limitations:

Despite the familiarity of simple flowcharting, visual notations don’t scale well in terms of representing program logic. Therefore, for dynamic or procedural problems, they tend to be used informally, as sketch or whiteboarding, or at the business analysis level (where the flowchart represents business logic, not detailed software). Dynamic processes also change more often than the static structures, and so must be updated more frequently.

More static structures, including data and class models and systems interactions, are still often represented visually and in the case of data models can be transformed from conceptual representations to physical schema.

However, any diagram, whether of a dynamic or static problem, can reach a level of density where it is no longer useful as a visual explanation. As diagrams become more complex, their audience narrows to those most familiar with them. Past a certain point, they exceed the limits of human visual processing and then are of little use to anyone.

This brings up broader concerns of the limits of human cognition; recent research shows that it is difficult for humans to hold more than four things in working memory — this is lower than previous estimates [204]. Diagrams with more than four to seven elements risk being dismissed as unusable.

Another issue with some diagrams is that they do not give a good sense of perspective or scale. This is sometimes seen in the Business Architecture practice of “capability mapping”. For example, suppose you see a diagram such as shown in Simple Capability Map.

Diagrams like this are common, but what does it mean that all the boxes are equally sized? Are there as many lawyers as sales people? Operations staff? It is not clear what the advantage is to putting information like this into a graphical form; no interactions are seen, and the eight areas could more easily be expressed as a list (or “catalog” in the terms we will introduce below). This brings us to the final problem listed above: visualizations rely on some common ground understanding. If boxes and lines are used for communication, their meaning should be agreed — otherwise, there is a risk of misunderstanding, and the diagram may do more harm than good.

Regardless of the pitfalls, many architecture diagrams are valuable. Whether drawn on a whiteboard, in Powerpoint™ or Omnigraffle™, or in a repository-based architecture tool, the visualization concisely represents a shared mental model on how the organizations will undertake complex activities. The diagram leverages the human preference for visual processing, accessing the powerful parallel processing of the visual cortex. Ultimately, the discussions and negotiations the architect facilitates on the journey to driving organizational direction are the real added value. The architect’s role is to facilitate discussions by abstracting and powerfully visualizing so that decisions are illuminated and understood across the team, or broader organization.

As the previous quote from the TOGAF standard indicates, architecture can elegantly be represented as:

For example, consider the image shown in Process and Function Diagram.

It can be read as saying that the “Quote to Cash” process depends on the following functions:

Notice that a matrix (see Process and Function Matrix) can be read in the same way.

“Quote to Cash”, which appeared as a chevron in the diagram, is now one of a list:

This list can be called a “catalog”. Similarly, there is another catalog of functions:

The functions appeared as rounded rectangles in the diagram.

There are pros and cons to each approach. Notice that in about the same amount of space, the matrix also documented the dependencies for two other processes and six other functions. The matrix may also be easier to maintain; it requires a spreadsheet-like tool, where the diagram requires a drawing tool. But it takes more effort to understand the matrix.

Maintaining a catalog of the concepts in a diagram becomes more and more important as the diagram scales up. Over time, the IT operation develops significant data by which to manage itself. It may develop one or more definitive portfolio lists, typically applications, services, assets, and/or technology products. Distinguishing and baselining high-quality versions of these data sets can consume many resources, and yet managing the IT organization at scale is nearly impossible without them. In other words, there is a data quality issue. What if the boxes on the diagram are redundant? Or inaccurate? This may not matter as much with a tight-knit team working on their whiteboard, but if the diagram is circulated more broadly, the quality expectations are higher.

Furthermore, it is convenient to have data such as a master lists or catalogs of processes, systems, functions, or data topics. We might also want to document various attributes associated with these catalogs. This data can then be used for operational processes, such as risk management, as we have discussed previously. For these reasons and others, Enterprise Architecture repositories emerge.

When we establish a catalog of architectural entities, we are engaging in master data management. In fact, the architectural concepts can be represented as a form of database schema (see A Simple Metamodel).

Material that we first saw in diagram form can be stored in a database. Systems that enable this are called Enterprise Architecture repositories. Their data schemas are often called metamodels.

Architecture repositories require careful management. A common anti-pattern is to acquire them without considering how the data will be maintained. The concepts in the repository can be subjective, and if it is intended that they be of high data quality, investments must be made. Some kind of registration process or decision authority must exist for the creation of (for example) a new, official “system” record. Misunderstandings and disagreements exist about the very meaning of terms like “system” or “technology”. (We discussed some of the general issues in Information Management, with the ontology problem.) Such issues are especially difficult when Enterprise Architecture repositories and metamodels are involved. Frequent topics:

And so on. We might expect that there would be industry standards clarifying such issues, and in some cases there are. In other cases, either there are no standards, or the standards are obsolete or conflicting.

Finally, there are a number of other systems that may interoperate with the architecture repository. The most important of these is the CMDB or CMS that underlies the ITSM tooling. These tools also need to know at least about systems and technologies and may be interested in higher-level concepts such as business capability. However, they usually do not include sophisticated diagramming capabilities or the ability to represent a system’s future state.

Other tools may include project management systems, portfolio management systems, risk management systems, service-level management systems, and others. Application and service master data, in particular, is widely used, and if the Enterprise Architecture repository is a System of Record for this data there will be many outbound interfaces.

Part of the challenge of any repository is what data to manage. How do we think more systematically about this? First, we need to understand why we want to assemble this data in a ready-to-query repository. There are two major reasons why we store data:

For either kind of data, you need to have an economic understanding of why you want it. Suppose you need to find out what applications you are running because you want to rationalize them. You could invest weeks of research into the question, costing perhaps tens of thousands of dollars worth of yours and others’ time, to create a one-time spreadsheet.

But what happens when there are multiple purposes for the data? You find out that the security group also wants a master list of applications and has been compiling a different spreadsheet, for example. What happens when the same engineers and managers are asked for the same data over and over again because there is no repository to maintain this organizational memory?

The challenge is, when does it make economic sense to pre-aggregate the data? The economic graph shown in Economic Value of the Enterprise Architecture Repository may assist in thinking about this.

The graph may be familiar to those of you who studied economics. On the left, you have the assumption of no architecture repository, and on the right, you have a comprehensive architecture repository. With a less comprehensive architecture repository, you are paying some cost in research and outage impacts. You also are incurring more risk, which can be quantified. On the other hand, with a comprehensive architecture repository, you incur more costs in maintaining it. You need processes that have a direct cost to operate, as well as imposing indirect costs such as the cost of delay (e.g., if updating the architecture repository slows down the release schedule).

But in the middle is a sweet spot, where you have “just enough” architecture repository data. This optimal architecture repository scope represents the real savings you might realize from instituting the architecture repository and the necessary processes to sustain it.

This is not a complete business case, of course. Your projected savings must be offset against the costs of acquisition and operations, and the remaining “benefit” needs to exceed your organization’s hurdle rate for investments.

From top to bottom, the data architecture (“what”) perspective includes concepts such as:

We covered data architecture in depth in Information Management.

Process architecture is concerned with why and how activities are performed. It includes the detailed, step-by-step understanding of activities, in a transparent way. From top to bottom, it includes concepts such as:

Importantly, processes cross organizations. An “onboard employee” process, as we have seen, may require participation from multiple organizations. We covered process management in depth in Coordination and Process.

The last column in the figure Simplified View of the ArchiMate Framework represents steady-state activities. “Hire Employee” is a process; “Manage Human Resources” is a capability. We do not necessarily know all the steps or details; we just know that if we ask the function or capability for some result, it can produce it. This perspective includes:

We discussed functional organization previously in Competency Area 9. Note that there is little consensus (and as of the time of writing much industry debate) around whether functions are the same as capabilities; this document sees them as at least similar. Capability is an important concept in Business Architecture, as it has emerged as the preferred concept for investment. We do not invest in data, or process, except as they are realized by a supporting capability. A comprehensive graphical depiction of “capabilities” may be used to help visualize portfolio investments, sometimes using green/yellow/red color coding — this is called "capability heat mapping”.

The Open Group Open Business Architecture (O-BA) Standard defines Business Architecture as: "The formalized description of how an organization uses its essential competencies for realizing its strategic intent and objectives."

The TOGAF Standard, Version 9.2 defines Business Architecture as: "A representation of holistic, multi-dimensional business views of: capabilities, end-to-end value delivery, information, and organizational structure; and the relationships among these business views and strategies, products, policies, initiatives, and stakeholders."

The TOGAF Series Guide: Business Models covers the Osterwalder Business Model Canvas extensively. In so doing, it highlights that the concept of the business model is of interest for Business Architects. Because of this, it is helpful to view Business Architecture as the component of Enterprise Architecture most concerned with the business model, in addition to the operating model.

More specifically, there are a number of concerns that Business Architecture includes:

from [50 p. 2]. The reader might notice some overlap with governance elements, which also include Information, Policies, and Organization.

On the other hand, we do not expect to see in Business Architecture the following:

Application, or application systems, like data, process, and capability, is a fundamental and widely-used architecture perspective, as well as a layer. It can be defined as "a fixed-form combination of computing processes and data structures that support a specific business purpose” [32 p. 125]. An application system is practically relevant, obtainable, and operable. (You can buy, or realistically build, one of these.)

Application architecture can have two meanings:

For this document, we will leave the architecture of a given application for solutions and software architecture. Application architecture is the interaction of multiple applications (which may include digital products and/or services, depending on organization terminology). In a complex, multi-product environment, application architecture tends to focus on the interfaces and interactions between the application systems. It is often a concern when systems are considered for retirement or replacement (for example, when a comprehensive ERP solution is brought in to replace several dozen home-grown applications).

Application architecture is also concerned with the application lifecycle, as covered at the start of this section.

Where Business Architecture intersects with the business model, technical architecture overlaps with actual engineering and operations. In particular, technical architecture tends to be concerned with:

Solutions architecture, especially, is a loose term. In general, it is restricted to one product, or a few products which are working together, as a “solution” to a business problem. Within that scope, it may incorporate concepts from infrastructure to Business Architecture. It also connotes more technical concerns.

The rapid evolution of software technology has fueled the growth of digital business. Following Internet giants’ lead, some enterprises from the old economy are framing themselves as tech companies; for example, Banco Bilbao Vizcaya Argentaria (BBVA): “If you want to be a leading bank, you have to be a technology company.”.

Internet giants did succeed at retaining the agility of startups while they grow at a fast pace and operate at a global scale. They paid special attention to loose-coupling and team autonomy and they learned how to master distributed computing at scale.

Since loose-coupling and distributed computing at scale are software architecture concerns, digital enterprises have to develop robust software architecture capabilities.

DDD has deeply influenced the software engineering discipline by shifting the attention to the domain. The most significant source of complexity of much software is not technical. It is in the domain itself, the activity or business of the user.

The premise of DDD is that:

DDD provides strategy patterns that help design loosely-coupled systems. The modular nature of a software system architected with DDD makes it cloud-native friendly by design. The code quality improves because domain concepts are made explicit.

The code is more readable (even by business people) and easier to maintain. Value objects drive a coding style that promotes immutability which makes distributed systems safer. Domain code is more immune from future technology obsolescence because it isolates and protects "domain" code from "technical" code which is more subject to technology obsolescence.

Marc Andreessen once wrote: "Software is eating the world." Evidence of this is the increasing scope of software technology usage. For example, IT infrastructure is becoming a software discipline. Infrastructure as Code and SDNs handle computing and networking resources as software objects. From a DDD perspective this opens domain modeling well beyond business into technology domains.

AI technologies such as Deep Machine Learning push the limits of automation. Software can now perform more and more activities that used to be exclusively performed by human beings such as driving a car or performing a medical diagnosis.

Software becomes a key component of complex product and service systems; for example, an automated parking or a self-service system. This has two implications:

Information architecture may mean the higher, more business-relevant levels of data architecture. However, the term also is used in relation to application architecture, in the sense of how the user understands the meaning and data represented by a website or application, or even just the navigation structure of a website.

In a digital world, aligning business and IT is no longer sufficient. You have to architect the enterprise in such a way that it can adapt to emerging customer and business needs.

The information systems of large organizations are becoming more complex which increases coordination efforts, raises failure rates of large transformation projects, or lowers the enterprise’s agility.

The CAS theory provides key concepts to help design systems that can evolve while complexity is maintained under control (i.e., co-evolution, interconnected autonomous agents, self-organization).

Enterprise Architecture needs to evolve toward an Adaptive Enterprise Model that facilitates its co-evolution with the environment and delegates decision-making to the lowest possible level.

The enterprise develops and delivers products and services that meet the ever-changing needs of customers and markets. Cross-functional features teams or squads led by product owners are given the autonomy to experiment and evolve products and services.

At the enterprise level, strategy defines the purposes that help align autonomous teams. Business models define how the enterprise will co-evolve within its eco-system. Operating models define how the required business capabilities will be supported.

The emergence of product platforms helps reuse capabilities that several products or services need. Product platforms accelerate the creation of new products and lower their development and production costs.

Figure: Toward an Adaptive Enterprise Model represents an evolution of the classical enterprise model. It complements it with concepts borrowed from Agile requirements, Lean Start-up and LPPD.

Let’s now zoom in on customer experience. In the old paradigm the enterprise would capture customer requirements and build products and services that meet those. The approach is mainly driven by market studies that try to "guess" what customers need.

In the digital world, guessing is replaced by observation and experimentation. The focus shifts to modeling the "Jobs to be Done" [61] by customers. The rapid development and experimentation of an MVP helps the enterprise confirm its direction or pivot to a different concept.

The art of architecting itself moves from a pure top-down role to a coaching role where capability modeling and modularity principles help define the Minimum Viable Architecture (MVA).

Figure: The Customer Experience’s Paradigm Shift illustrates the shift toward an outside-in approach. It has the following benefits:

Evidence of Notability

There are a wide variety of architectural practices and approaches frequently addressed and debated within industry. They are codified within the TOGAF standard [279] as well as other literature.

Limitations

Architecture often manifests as an impulse to plan in advance; however, in complex situations it is difficult to plan before a full understanding of the problem is achieved. This is a well-known problem in software engineering.

Related Topics

Cost of delay is a real and often overlooked issue, in understanding the net value of architecture. But it is only a factor and does not eliminate the value proposition of architecture. If the cost of delay is only a few hundred dollars a month, but the risk or technical debt represents millions, then the delay may be appropriate. Don Reinertsen, who has done more than anyone to promote the idea of cost of delay, emphasizes that all decision-making must take place within an economic framework and that means that the other architectural impact factors on organization value must also be considered [230], Chapter 2.

Documentation has been a core concern of the Agile movement, being mentioned in one of the four core principles of the Agile Manifesto:

"Working software over comprehensive documentation.” [8]

When documentation primarily takes the form of secondary artifacts, it is appropriate to question the need for it. “The code is the documentation”, some will argue. While it is true that good coding practices result in easier-to-understand (and maintain) source code, the code cannot be the only documentation. As Ruth Malan notes:

…​ for systems of sufficient scope and complexity to warrant teams (of teams) working on (incremental) implementation and evolution, the sheer mass of code can make it hard to discover the essential structure from bottom-up decisions made entirely through the medium of code. [186]

In terms of systems theory, a complex software system has emergent behavior, not obvious from just looking at its components. Because the system’s behavior can’t be reduced to its pieces, “self-documenting code” can only go so far. The behavior of the assembled components as a system needs to be represented somehow, in a way that transcends the mere mechanics of the pieces. Abstraction is necessary to understand and communicate emergent behavior, and this leads inevitably to visual representation. Without some attention to documenting overall context and systemic intent and behavior, the effectiveness of the overall human/computer system degrades. For example, Alistair Cockburn reports that the Chrysler Comprehensive Compensation project, one of the first widely reported Agile projects, was eventually halted, and:

…​ left no archived documentation …​ other than two-sentence user stories, the tests, and the code. Eventually, enough people left that the oral tradition and group memory were lost. [65 pp. 41-43]

In short, failure to sustain a shared mental model of a complex system is a risk that may result in loss of that system’s value.

Agile and DevOps are software development-centric, and have transformed that world. However, digital organizations don’t always build everything. There is a complex web of supplier relationships even for organizations with robust software development capabilities, and many organizations would still prefer to “buy rather than build”. Software may be consuming the world, but that doesn’t mean everyone employs — or should employ — software developers. Agile has not had a primary focus on sourcing, and evaluating commercial software is not a common Agile topic.

Suppose you have an idea for a digital product, and you know that you will be (at least in part) assembling complex services/products produced by others? Suppose further that these provided services overlap (the providers compete)? You need to carefully analyze which services you are going to acquire from which provider. You will need a strategy, and who is it that analyzes these services and their capabilities, interfaces, non-functional characteristics, and makes a final recommendation as to how you are going to bring them all into one unified system?

It is easy to say things like: “the teams get to define their own architecture”, but at some point, the enterprise must reckon with the cost of an overly diverse supplier base. This is a very old topic in business, not restricted to IT. At the end of the day, supplier and sourcing fragmentation costs real money. Open source, Commercial-Off-The-Shelf (COTS), cloud, in-house …​ the options are bewildering and require experience. In a sense, the supplier base itself is an inventory, subject to aging and spoilage. (We can consider this another way of understanding technical debt.) A consistent evaluation approach is important (preferably under an economic framework; see Reinertsen & Hubbard). And at some point, product development teams should not have to do too much of their own R&D on possible platforms for their work.

The Agile Manifesto is well known for saying: “The best architectures, requirements, and designs emerge from self-organizing teams” [8]. This is one of the more frequently discussed Agile statements. Former Netflix CTO Adrian Cockcroft has expressed similar views (quote above).

A key question is whether “architecture” is considered at the single product or multi-product level. At the single product level, collaborative teams routinely develop effective software architectures. However, when multiple products are involved, it is hard to see how all the architectural value scenarios are fulfilled without some investment being directed to the goals of cross-product architectural coordination. It helps when rules of the road are established; both Amazon and Netflix have benefited from having certain widely-accepted platform standards, such as “every product communicates through APIs”. Netflix had for a long time a long-term commitment to Amazon cloud services; it was not acceptable for teams there to decide on a whim to deploy their services on Google Compute Engine™ or Microsoft Azure™, so at least in that sense Netflix has an architecture. The question gets harder when layered products and services with complex lifecycle interactions are involved.

Microservices can reduce the need for cross-team coordination, but coordination needs still do emerge. For example, Mike Burrows of Google provides a detailed description of the Chubby lock service [49], which is a prototypical example of a broadly-available internal service usable by a wide variety of other products.

The purpose of a lock service is to “allow its clients to synchronize their activities and to agree on basic information about their environment”. Chubby was built from the start with objectives of reliability, availability to a “moderately large set of clients”, and ease of understanding. Burrows notes that even with such a cohesive and well-designed internal service, they still encounter coordination problems requiring human intervention. Such problems include:

Because of this, the Chubby team (at least at the time of writing the case study) instituted a review process when new clients wished to start using the lock manager. In terms of Competency Area 7, this means that someone on the product team needed to coordinate the discussions with the Chubby team and ensure that any concerns were resolved. This might conceivably have involved multiple iterations and reviews of designs describing intended use.

Thus, even the most sophisticated microservice environments may have a dependency on human coordination across the teams.

One principle throughout this document has been “respect the team”, because true product value originates there. If teams are constantly fragmented and their cohesion degraded by enterprise operating models and governance mandates, their ability to creatively solve business problems is hampered. Command and control replace emergence, motivation declines, and valuable creativity is lost. Enterprise Architecture must first and foremost protect the precious resource that is the high-performing, collaborative, creative team. As we have discussed, imposing multiple governance checkpoints itself adds risk. And while it is inevitable that the team will be subject to organization-wide mandates, they should be given the benefit of the doubt when autonomy collides with standardization.

When Enterprise Architecture takes on true Business Architecture questions, including how digital capabilities are to be enabled and enhanced, Agile insights become an input or kind of requirement to Business Architecture. What capabilities require high-performing, cross-functional teams? What capabilities can be supported by project-based temporary teams? And what capabilities should be outsourced? The more valuable and difficult the work, the more it calls for the careful development of a common mental model among a close-knit team over time. Driving organizational capability investment into long-running team structures becomes a strategy that organizational architects should consider as they develop the overall organizational portfolio.

Architecture adds value through constraining choices. This may seem counterintuitive, but the choice is often between re-using a known existing platform or engaging in risky R&D of alternatives. R&D costs money, and itself can impose a delay on establishing a reliable digital pipeline. But ultimately, the fundamental objective remains customer and product discovery. All other objectives are secondary; without fulfilling customer needs, architectural consistency is meaningless. Optimizing for the fast creation of product information, tested and validated against operational reality, needs to be top of mind for the architect.

The digital pipeline ultimately is a finely-tuned tool for this creation of information. It, itself, has an architecture: business, application, and technical. It operates within an economic framework. To understand the architecture of the digital pipeline is in a sense to understand the “architecture of architecture”.

As we have discussed above, architecture, like staff functions generally, is in part a coordination mechanism. It collects and curates knowledge and sustains the organization’s understanding of its complex systems. Architecture also identifies gaps and informs the investment process, in part through collecting feedback from the organization.

If architecture’s fundamental purpose is enabling the right emergent behavior, there are still questions about how it does so. Architecture adds value in assisting when:

As a coordination mechanism, it can operate in various ways including planning, controlling, and collaborating. Each may be appropriate for a given challenge or situation. For example, different approaches are required depending on whether the product challenge is flower or cog. A flower is not engineered to fill a gap. A cog is. Market-facing experiments need leeway to pivot, where initiatives intended to fill a gap in a larger system may require more constraints and control. And how do architects know there is a gap? It should be an hypothesis-driven process, that needs to establish that there is a valuable, usable, feasible future state.

As an executing capability, architecture operates in various ways:

Ideally, planning and analysis occur “upstream” of the creation of a product team. In that guise, architecture functions as a sort of zoning or planning authority — “architecture” is not a process or organization directly experienced by the product team. In this ideal, there is no conflict with product teams because once the team is formed, the architect’s job is done. However, this assumes that all the planning associated with launching a new product or capability was done correctly, and this itself is a kind of waterfall assumption. Some form of feedback and coordination is required in multi-product environments.

It is in the “governance and approval” kind of activity that conflict is most likely to emerge. Cadence and synchronization (e.g., coordination strategies) with the potential to block teams from pursuing their mission should be very carefully considered. If there is a process or a queue of architecture approvals, it at least should be operated on cost of delay of the work it is blocking. And more generally, across the organization, the process should be tested against an economic model such as establishing a nominal or portfolio-level cost of delay. Like other processes, architecture itself can be assessed against such a baseline.

Queued approvals are only one way of solving issues. A rich and under-utilized approach is using internal market-type mechanisms, where overall rules are set, and teams make autonomous decisions based on those rules. Don Reinertsen, in the Principles of Product Development Flow, discusses how Boeing implemented distributed decision-making through setting trade-off rules for cost and weight. Rather than constantly routing design approvals through a single control point, Boeing instead set the principle that project managers could “purchase” design changes up to $300 per unit, to save a pound of weight. As Reinertsen notes:

The intrinsic elegance of this approach is that the superiors didn’t actually give up control over the decision. Instead, they recognized that they could still control the decision without participating in it. They simply had to control the economic logic of the decision. [230 p. 42]

One particular work product that architects often are concerned with is documentation. The desire for useful documentation, as mentioned above, reflects architecture’s goals of curating a common ground for collaboration. As Bente notes: “In an Agile project, explicit care must be taken to ensure proper documentation; for example, by stating it as part of the condition of satisfaction of a user story or in the definition of done.” [28 p. 170]

Toyota Kata was discussed in Organization and Culture. In Lean Enterprise, Jez Humble et al. argue that it can provide a useful framework for architecture objectives. Toyota Kata emphasizes end-state goals (“target conditions”) and calls for hands-on investigation and response by participating workers, not consultants or distant executives. Architecture can benefit by understanding “gaps” in the sense of Toyota’s target conditions, and then support teams in their collaborative efforts to understand and achieve the desired state. The architectural impact model can assist in thinking through suitable target conditions for architecture:

Keeping the target operating condition specific is preferable. When architecture scopes problems too broadly, the temptation is to undertake large and risky transformation programs. As an alternative, Humble suggests the "strangler pattern”, proposed by Martin Fowler in 2004 [102]. This pattern uses as a metaphor Australian “strangler” vines that grow around trees until the original tree dies, at which point the strangler vine is now itself a sturdy, rooted structure.

To use the strangler pattern is not to replace the system all at once, but rather to do so incrementally, replacing one feature at a time. This may seem more expensive, as it means that both the old and new systems are running (and cost savings through sunsetting the old system will be delayed). But the risk of replacing complex systems is serious and needs to be considered along with any hoped-for cost savings through replacement. Humble and Molesky suggest:

The strangler pattern is proven in practice. Paul Hammant provides a large number of strangler pattern case studies, including:

and others [120].

Of course, there are other ways architecture might add value beyond system replacement, in which case the strangler pattern may not be relevant. In particular, architects may be called on to closely collaborate with product teams when certain kinds of issues emerge. This is not a governance or control interaction; it is instead architecture as a form of shared consulting “bench” or coordination mechanism. Not every product team needs a full-time architect, the reasoning goes, so architects can be assigned to them on a temporary basis; e.g., for one or a few sprints, perhaps of the technical “spike” (discovery/validation/experimentation) variety.

Finally, how do we evaluate architecture outcomes? If an organization adopts an experimental, Toyota Kata approach, it may find that architecture experiments run on long time horizons. Maintaining an organizational focus on value may be challenging, as the experiments don’t yield results quickly. Curating a common ground of understanding may sound like a fine ideal, but how do we measure it?

First, the concept of Net Promoter Score is relevant for any service organization, internal or external. Its single question: “Based on your experience, on a scale of 1-10 would you recommend this product or service to a friend?” efficiently encapsulates value in a single, easy to respond to query.

As digital pipelines become more automated, it may be possible to evaluate their digital exhaust impact on architecture services:

In a world of increasing connectivity and automation, there is no reason for architects in the organization to lack visibility into the consequences of their recommendations. Ultimately, if the cost of operating the coordination mechanism that is architecture exceeds the value it provides, then continuing to operate it is irrational.

Evidence of Notability

Agile, DevOps, and architecture often come into contact and even conflict. This friction carries many consequences for organizations wishing to digitally transform, yet not abandon the benefits of architecture.

Limitations

The intersection of Agile and architecture is most significant in organizations that are performing their own digital R&D (i.e., software development).

Related Topics