Managing Digital: Concepts and Practices

We saw in Chapter 10 how governance emerges, as a response to scale and growth, and the concerns for risk and assurance in the face of increasing pressures of the external environment.

Architecture has a comparable emergence story, but for somewhat different reasons. Consider the quote from Elihu Root, above. Root was tasked with re-organizing the U.S. Army at the turn of the 20th century. There had been some embarrassing failures of coordination and organization, and it was clear that something was missing in the U.S. military: a centralized coordination function responsible for planning and logistics.

These lessons had also been learned by the Austrian, Prussian, and French militaries.

In previous ages, nobles would assemble and fight for their King, but their armies were poorly coordinated, and disputes over strategy would often arise. Each noble’s army would have its own quartermaster, couriers, intelligence, supply chains, and the like. This was both inefficient, and ineffective: overall plans of battle often could not be made or executed, and military operations would be bungled.

As fighting wars became (unfortunately) a larger and larger scale endeavor, the need to centralize certain capabilities became more and more obvious. This culminated in the Napoleonic wars of the late 1700s and early 1800s with the creation of “general staffs” that were responsible for coordination of planning and execution across the increasingly complex military operations. The HQ “staff” and its shared services were deliberately distinguished from the warfighting “line.” The Austrian field marshall Franz Moritz Graf von Lacy (see Franz Moritz Graf von Lacy ^[3]) was one of the pioneers of this new style of organization, which was soon copied by the French army under Napoleon Bonaparte.

For example, the role of Napoleon’s staff officer Pierre-Joseph Bourcet under Napoleon is described thus:

On every occasion when an important decision had to be made Bourcet would write a memorandum in which he analyzed the situation and set forth in detail, with full explanations and reasons, the course which seemed to him best. In very many cases, his suggestions were adopted and were usually justified by success, and when they were rejected the results were seldom fortunate. [152]

What do obscure French and Austrian military officers have to do with today’s digital organizations? They are analogous to the line/staff as the basis for large organizations of all kinds. As Christian Millotat (and many others) have noted, "[m]any elements that have become integral parts of managerial economics and organizing sciences can be traced back” to military staff systems [186], p.7. These include:

Staff functions in the enterprise include planning, coordination, and operations; broadly speaking, and with key differences depending on the industry, the following are considered “staff":

While the following are considered “line” (analogous to the warfighting units in the military):

Enterprise architecture has as a key part of its mission the task of collecting and combining knowledge to support decision-making. Therefore, an Enterprise Architecture organization can be seen as a form of staff organization. Most often it is seen as a specialized staff function within the larger staff function of IT management, and with the increased role of digital technology, there are corresponding pressures to “move Enterprise Architecture out of IT” as we’ll discuss below.

The classic line/staff division is a powerful concept, pervasive throughout organizational theory. But it has important limitations:

In the history of line versus staff relations, we see tensions similar to those between Enterprise Architecture and advocates of Agile methods. The challenges, debates, and conflicts have only changed in their content, but not their essential form.

But in many cases, centralizing staff expertise and the definition of acceptable practices for a domain is still essential. See the discussion of matrix organizations and even feature versus component teams.

In terms of overall positioning, Enterprise Architecture is often portrayed as mediating between strategy and portfolio management (see EA context, based on Ross., derived from [226 p. 10], fig 1-2).

Notice the distinction between Enterprise Architecture as a capability and as an artifact. The practice of Enterprise Architecture is not the same as the actual Enterprise Architecture. For the purposes of this textbook, we define Enterprise Architecture’s concerns as essentially the enterprise operating model: process, data, organizational capabilities, and systems.

Scott Bernard (author of An Introduction to Enterprise Architecture) claims that: [without Enterprise Architecture] “…​leadership will not have the ability to generate clear, consistent views of the overall enterprise on an ongoing basis, they won’t be able to compare business units effectively, and the locus of power for planning and decision-making will be at the line-of-business, program, and/or system owner levels with significant differences in how things are done and high potential for overlapping or duplicative functions and resources.” [22], introduction.

Now, truth be told, corporations compare business units all the time, on profit and loss and other enterprise metrics, without an architecture. If an enterprise is a holding company, by definition, it is not seeking a common operating model. So, it is important to understand the role of Enterprise Architecture within the context of the enterprise strategy and business model, which remain distinct.

But, assuming that some shared vision, intelligence, economies of scale, and shared services are part of the business model, the concept of an operating model is a powerful tool for categorizing critical information, and identifying redundancies, overlaps, and potential synergies.

One of the most frequently used visualizations of Enterprise Architecture’s concerns is the Zachman Framework (see A variation on the Zachman framework ^[4]).

We were exposed to the data modeling progression from conceptual to logical to the physical data model in Chapter 11. The Zachman framework generalizes this progression to various views of importance to organizations, as shown in the columns:

Overall, the Zachman framework represents the range of organizational operating model concerns well. Certainly, sustaining a large and complex organization requires attention to all its concerns. But what good does it do to simply document the contents of each cell? Such activity needs to have relevance for organizational planning and strategy; otherwise, it is just waste.

It’s reasonable to associate the Business Model Canvas with strategy, and the Zachman framework with the Enterprise Architecture as an artifact (see Business model versus operating model).

This distinction helps us position the Enterprise Architecture group with respect to key partner organizations:

Organizational strategy

The operating model needs to support the business model, and so, therefore, Enterprise Architecture needs a close and ongoing relationship with organizational strategists, whether they are themselves line or staff. Defining digital strategies is a challenging topic; we have touched on it in Chapter 1, Chapter 4, and Chapter 8. Further discussion at the enterprise level will be deferred to a future edition of this book.

Portfolio and investment management

Architecture needs to be tied to the organization’s investment management process. This may be easier said that done, given the silos that exist. As Scott Bernard notes, “Enterprise architecture tends to be viewed as a hostile takeover by program managers and executives who have previously had a lot of independence in developing solutions for their own requirements” [22], case study scene 1.

Many organizations have a long legacy of project-driven development, in which the operational consequences of the project were too often given short shrift. The resulting technical debt can be crippling. Now that there is more of a move towards “you build it, you run it” the operability aspects of systems are (perhaps) improving. However, ongoing scrutiny and management are still needed at the investment front end, if the enterprise is to manage important objectives like vendor leverage, minimizing technical debt, reducing investment redundancy, controlling the security perimeter, and keeping skills acquisition cost efficient (more on this below in the section on Enterprise Architecture value).

Portfolio management is discussed in depth in a subsequent chapter section.

Infrastructure and operations

Finally, the Enterprise Architecture group often has a close relationship with infrastructure and operations groups. This is because in organizations where operations is a shared service, the risks, and inefficiencies of technical fragmentation are often most apparent to the operations team. In organizations where operations is increasingly distributed to the application teams (“you build it, you run it”) the above may be less true.

Other staff organizations that may develop close relationships with Enterprise Architecture include vendor management and sourcing, risk management, compliance, and security. Notice that some of these have strong governance connections (although we do not consider “governance” itself to be an organization, which is why it was not included in the discussion above).

We have covered the concept of cost of delay previously, at the team and product level. However, this powerful concept can also be applied at higher levels. cost of delay is not a concept familiar to most architects. It poses two important challenges:

As we’ve discussed previously, the definition of cost of delay is intuitive. It is the opportunity cost of not having a given product or service available for use: the foregone revenues, the cost of the workarounds and inefficiencies. If the architecture process becomes the critical path for a product or service’s release (a common experience), then the Architecture process is responsible for that product’s or services’ cost of delay.

The cost of delay can take various forms, some of them significant. For example: suppose there is a need to demonstrate a product to key clients at a trade show. This could be the company’s best opportunity to develop business; the sales team estimates $12 million in funnel opportunities based on previous experience that should result in at least $2 million in sales in the year, with projections of another $1 million in maintenance and renewals. However, if the product is not ready, these benefits will not materialize. If everything else is ready, but the Architecture process is delaying product readiness, then the Architecture process is incurring $2 million in the cost of delay. This is bad. The Architecture process is clearly impacting significant business objectives and revenue.

But the question still needs to be asked, “what benefits do we receive from having an Architecture process?” We discussed such benefits above. Are these benefits adding up to $2 million a year? No? Then your Architecture process does not make good economic sense. On the other hand, what if the architects were kept out of the picture, and the product team chooses an untested technology, instead of re-using a well known and reliable approach already proven for that company? What if that decision were the cause of missing the trade show? What if it can be shown that re-usable components identified as architecture standards were increasing the speed of delivery, and reducing the cost of delay, because they are reducing the need for product teams to perform risky (and yet redundant) Research and Development activities?

Quantifying these benefits across a portfolio is difficult, but should be attempted. cost of delay can and should be calculated at the portfolio level, and this can provide “enterprise-level decision rules” that can help an organization understand the cost and value of operating model changes [219 pp. 35-38] including instituting processes (such as Change Management or Technology Lifecycle Management), or even the establishment of Enterprise Architecture itself.

We touched on technical debt in Chapter 3’s discussion of refactoring. Technical debt is a metaphor first introduced by Ward Cunningham [75] in the context of software, widely discussed in the industry. It can be applied more broadly at the portfolio level, and in that sense, is sometimes discussed in Enterprise Architecture. Debt exists in the form of obsolete products and technologies; redundant capabilities and systems; interfaces tightly coupled where they should be loose and open, and many other forms.

Technical debt, like the cost of delay, can and should be quantified. We’ll discuss approaches to that in the chapter section on portfolio management.

We’ve often referred to the concept of common ground through this book. The architecture supports common ground understanding at scale, by curating a shared mental model for the organization. In doing so, it enables the “right emergent behaviors” (as Adrian Cockcroft suggests). It also enables communication across diversity and may improve staffing flexibility and mobility among teams.

Dan Moody notes [188]:

Visual representations are effective because they tap into the capabilities of the powerful and highly parallel human visual system. We like receiving information in a visual form and can process it very efficiently: around a quarter of our brains are devoted to vision, more than all our other senses combined [63]. In addition, diagrams can convey information more concisely [27] and precisely than ordinary language [8, 68]. Information represented visually is also more likely to be remembered due to the picture superiority effect [38, 70] …​Visual representations are also processed differently: according to dual channel theory [80], the human mind has separate systems for processing pictorial and verbal material. Visual representations are processed in parallel by the visual system, while textual representations are processed serially by the auditory system…​ .

As the above quote shows, there are clear neurological reasons for diagramming as a communication form. To expand a bit more on the points Dan Moody is making:

Human vision uses parallel processing. This means that a given image or visual stimulus is processed by many neurons simultaneously. This is how we can quickly recognize and act on threats, such as a crouching tiger (as in Fast recognition means survival ^[7]).

A large percentage of our brain is devoted to visual processing. You will see figures quoted from 25% to 66% depending on whether it’s “pure” visual tasks or vision-driven tasks involving other brain areas.

The old saying "a picture is worth a thousand words" is consistent with the science. Diagrams can be both faster and more precise at conveying information; however, this has limits.

Finally, pictures can be more memorable than words.

Architects and architecture are known for creating diagrams -— abstract graphical representations of complex systems. The first known instance of applying graphical techniques to a digital problem was in 1947 (see The first software flowchart ^[8], and visual notations have evolved along with the field of computing ever since. Notable examples include

(We touched on data modeling in Chapter 11). We will examine the ArchiMate® Modeling Language, a standard of The Open Group, for a current and widely used notation, in more detail in a future chapter section.

Research at Microsoft suggests that developers use diagrams for four purposes:

They argue “diagrams support communicating, capturing attention and grounding conversations [4]. They reduce the cognitive burden of evaluating a design or considering new ideas [13]” [58].

But visual notations have been problematic in the Agile community; as Fowler notes in his essay Is Design Dead:

[Agile method eXtreme Progamming] de-emphasizes diagrams to a great extent. Although the official position is along the lines of “use them if they are useful," there is a strong subtext of “real XPers don’t do diagrams"…​ I think the issue comes from two separate causes. One is the fact that some people find software diagrams helpful and some people don’t. The danger is that those who do think that those who don’t should do and vice-versa. Instead, we should just accept that some people will use diagrams and some won’t. [98]

There is no question that some IT professionals, including perhaps some of the most skilled software engineers, find little of use in diagrams . As Martin Fowler says, “people like Kent [Beck, eXtreme Programming originator] aren’t at all comfortable with diagrams. Indeed, I’ve never seen Kent voluntarily draw a software diagram in any fixed notation.” However, it seems likely that Kent Beck and others like him are members of a programming elite, with a well-honed mental ability to process source code in its “raw” form.

However, if we’re building systems to be operated and maintained by humans, it would seem that we should support the cognitive and perceptual strengths of humans. Because diagrams are more readily processed, they are often used to represent high-level system interactions — how a given service, product or application is related to peer systems and services. Building such depictions can be helpful to fostering a shared mental model of the overall system objectives and context. The more complex and highly scaled the environment, the more likely one will encounter such artifacts as a means to creating the mental model.

The strength of human visual processing is why we will (probably) always use graphical representation to assist in the building of shared mental models. Specialists in the syntax and semantics of such designs will therefore likely continue to play a role in complex systems development and maintenance. Currently, if one seeks to hire such a specialist, one recruits some kind of architect — that is the professional role with the skills.

Note that flowcharts, data models, and other such diagrams tend to be associated more with the idea of “solutions” or “software” architecture. We’ll cover the architecture domains in the next chapter section, including examples of Business Architecture diagrams.

Visualization has a number of limitations:

Despite the familiarity of simple flowcharting, visual notations don’t scale well in terms of representing program logic. Therefore, for dynamic or procedural problems, they tend to be used informally, as sketch or whiteboarding, or at the business analysis level (where the flowchart represents business logic, not detailed software). Dynamic processes also change more often than the static structures, and so must be updated more frequently.

More static structures, including data and class models and systems interactions, are still often represented visually and in the case of data models can be transformed from conceptual representations to physical schema.

However, any diagram, whether of a dynamic or static problem, can reach a level of density where it’s no longer useful as a visual explanation. As diagrams become more complex, their audience narrows to those most familiar with them. Past a certain point, they exceed the limits of human visual processing and then are of little use to anyone.

For example, the illustration in Complex diagram ^[11], while intimidating, is likely useful to those who work with and study it. It would take some familiarization.

However,the diagram in Another complex diagram ^[12] is essentially unusable, as visually tracing any given line is too difficult, and it would be easy to mistakenly identify one bubble as dependent on another:

This may seem like an obvious critique, but architectural diagrams of similar complexity and unusability have too often been produced. This brings up broader concerns of the limits of human cognition; recent research shows that it is difficult for humans to hold more than 4 things in working memory -— this is lower than previous estiamates [192]. Diagrams with more than 4-7 elements risk being dismissed as unusable.

Another issue with some diagrams is that they do not give a good sense of perspective or scale. This is sometimes seen in the Business Architecture practice of “capability mapping.” For example, suppose you see a diagram such as shown in Simple capability map.

Diagrams like this are common, but what does it mean that all the boxes are equally sized? Are there as many lawyers as sales people? Operations staff? It’s not clear what the advantage is to putting information like this into a graphical form; no interactions are seen, and the eight areas could more easily be expressed as a list (or “catalog” in the terms we’ll introduce below). This brings us to the final problem listed above: visualizations rely on some common ground understanding. If boxes and lines are used for communication, their meaning should be agreed upon — otherwise, there is a risk of misunderstanding, and the diagram may do more harm than good.

Regardless of the pitfalls, many architecture diagrams are valuable. Whether drawn on a whiteboard, in Powerpoint or Omnigraffle, or in a repository-based architecture tool , the visualization concisely represents a shared mental model on how the organizations will undertake complex activities. The diagram leverages the human preference for visual processing, accessing the powerful parallel processing of the visual cortex. Ultimately, the discussions and negotiations the architect facilitates on the journey to driving organizational direction are the real added value. The architect’s role is to facilitate discussions by abstracting and powerfully visualizing so that decisions are illuminated and understood across the team, or broader organization.

As the above quote from the TOGAF standard indicates, architecture can elegantly be represented as:

For example, consider the image shown in Process and function diagram.

It can be read as saying that the “Quote to Cash” process depends on the following functions:

Notice that a matrix (Process and function matrix) can be read in the same way.

“Quote to cash,” which appeared as a chevron in the diagram, is now one of a list:

This list can be called a “catalog.” Similarly, there is another catalog of functions:

The functions appeared as rounded rectangles in the diagram.

There are pros and cons to each approach. Notice that in about the same amount of space, the matrix also documented the dependencies for two other processes and six other functions. The matrix may also be easier to maintain; it requires a spreadsheet-like tool, where the diagram requires a drawing tool. But it takes more effort to understand the matrix.

Maintaining a catalog of the concepts in a diagram becomes more and more important as the diagram scales up. Over time, the IT operation develops significant data by which to manage itself. It may develop one or more definitive portfolio list, typically applications, services, assets, and/or technology products. Distinguishing and baselining high-quality versions of these data sets can consume many resources, and yet managing the IT organization at scale is nearly impossible without them. In other words, there is a data quality issue. What if the boxes on the diagram are redundant? Or inaccurate? This may not matter as much with a tight-knit team working on their whiteboard, but if the diagram is circulated more broadly, the quality expectations are higher.

Furthermore, it is convenient to have data such as a master lists or catalogs of processes, systems, functions, or data topics. We might also want to document various attributes associated with these catalogs. This data can then be used for operational processes, such as risk management, as we have discussed previously. For these reasons and others, Enterprise Architecture repositories emerge.

When we establish a catalog of architectural entities, we are engaging in master data management. In fact, the architectural concepts can be represented as a form of database schema (see A simple metamodel).

Thus, material that we first saw in diagram form can be stored in a database. Systems that enable this are called Enterprise Architecture repositories. Their data schemas are often called metamodels.

Architecture repositories require careful management. A common anti-pattern is to acquire them without considering how the data will be maintained. The concepts in the repository can be subjective, and if it is intended that they be of high data quality, investments must be made. Some kind of registration process or decision authority must exist for the creation of (for example) a new, official “system” record. Misunderstandings and disagreements about the very meaning of terms like “system” or “technology.” (We discussed some of the general issues in Chapter 11, with the ontology problem). Such issues are especially difficult when Enterprise Architecture repositories and metamodels are involved. Frequent topics:

And so on. One might expect that there would be industry standards clarifying such issues, and in some cases there are. In other cases, either there are no standards, or the standards are obsolete or conflicting.

Finally, there are a number of other systems that may interoperate with the architecture repository. The most important of these is the configuration management database (CMDB) or system (CMS) that underlies the IT service management tooling. These tools also need to know at least about systems and technologies and may be interested in higher-level concepts such as business capability. However, they usually do not include sophisticated diagramming capabilities or the ability to represent a system’s future state.

Other tools may include project management systems, portfolio management systems, risk management systems, service level management systems, and others. Application and service master data, in particular, is widely used, and if the Enterprise Architecture repository is System of Record for this data there will be many outbound interfaces.

Part of the challenge of any repository is what data to manage. How do we think more systematically about this? First, we need to understand why we want to assemble this data in a ready-to-query repository. There are two major reasons why we store data:

For either kind of data, you need to have an economic understanding of why you want it. Suppose you need to find out what applications you are running because you want to rationalize them. You could invest weeks of research into the question, costing perhaps tens of thousands of dollars worth of yours and others’ time, to create a one-time spreadsheet.

But what happens when there are multiple purposes for the data? You find out that the security group also wants a master list of applications and has been compiling a different spreadsheet, for example. What happens when the same engineers and managers are asked for the same data over and over again because there is no repository to maintain this organizational memory?

The challenge is, when does it make economic sense to pre-aggregate the data? The economic graph shown in Economic value of Enterprise Architecture repository may assist in thinking about this.

The graph may be familiar to those of you who studied economics. On the left, you have the assumption of no architecture repository, and on the right, you have a comprehensive architecture repository. With a less comprehensive architecture repository, you are paying some cost in research and outage impacts. You also are incurring more risk, which can be quantified. On the other hand, with a comprehensive architecture repository, you incur more costs in maintaining it. You need processes that have a direct cost to operate, as well as imposing indirect costs such as the cost of delay (e.g.,if updating the architecture repository slows down the release schedule).

But in the middle is a sweet spot, where you have “just enough” architecture repository data. This optimal architecture repository scope represents the real savings you might realize from instituting the architecture repository and the necessary processes to sustain it. This is not a complete business case, of course. Your projected savings must be offset against the costs of acquisition and operations, and the remaining “benefit” needs to exceed your organization’s hurdle rate for investments.

A simple example: Let’s say that you have identified a set of use cases indicating $250,000 maximum savings from an accurate and optimally scoped architecture repository. The implementation is projected to cost $200,000 over 3 years, for net benefits of $50,000. This means that you have an ROI of 25% ($50k/$200k). You can’t get 25% return on investment in the stock market, so the CFO sees this as a good use of money. On the other hand, if you only are projecting $15,000 in net benefits, for an ROI of 8%, the CFO may well say, “I can do better with other projects, or by leaving our money in the stock market.”

Of course, estimating benefits such as reducing redundant research are not simple. The book How to Measure Anything by Doug Hubbard [127] is a very useful resource for these kinds of problems. You may need to consider using techniques such as Monte Carlo Analysis for business benefits that are more probabilistic. But you do not need to throw up your hands and say it’s all just “intangible,” successfully making challenging business cases of this nature is possible.

A large electronics retailer purchases a smaller chain of stores specializing in vinyl records and CDs, just as the market for these are declining. The decision is made to integrate all the systems: Point of Sale, inventory, HR & Payroll, etc., in the interest of efficiency and rationalization.

Work progresses on this effort for nearly a year, and then the surprising news is announced: the newly acquired company is to be sold off! All the work that went into rationalizing the systems is wasted, and the independent operating model for the smaller chain has to be completely restored. Clearly, in this case, rationalization was not rational.

A large financial institution had allowed its computer systems to become fragmented. Acquisitions and independence among the lines of business meant that if a customer moved, they might need to contact customer service repeatedly to change their address. This was a failure of master data management; there was no clear system of record for the customer address.

While we first discussed this kind of issue in Chapter 11, solving the problem requires more than data architecture. Establishing one data source as “master” requires considerable engineering in such an environment: there are capacity, security, interface, and business process concerns to work through, that in this particular case took tens of millions of dollars of investment, in response to increasing customer unhappiness.

And once a system of record is established and the data interfaces have been constructed reflecting its status, its role needs to be preserved. Independent teams may once again start to master their own customer data, “just for now,” because perhaps accessing the system of record seems to be too much work. Avoiding such scenarios is why some architecture organizations institute project reviews.

In 2006, Airbus was in the midst of developing the gigantic A380 aircraft. Because of the scale of the effort, multiple teams of engineers in different countries were working on the effort. As William Ulrich says of the German and French teams, “having failed to coordinate efforts to wire the world’s largest plane, they were surprised to learn that they had used different approaches to designing the A380’s wiring system” [271], p. 32. This was because the teams were using different software for wiring design of the critical wiring design software [61]. The use of different software to fulfill the capability resulted in a year’s delay, a $4.5bn order loss to Boeing, and a 26% drop in Airbus stock. ^[13]

Henrik Kniberg worked on the PUST project for the Swedish national police, which was a project to deploy laptops to the Swedish police. Because of its urgency, it was allowed to use Lean and Agile techniques, and received favorable media coverage and was generally deemed a success [158].

Unfortunately, a drive for technology “rationalization” resulted in the re-platforming of the system on a commercial platform; the Siebel CRM system [157]. Furthermore, an open loop waterfall method was used to deliver the system. Kniberg reports that the result has been an unsuccessful rollout, unhappy users, and cost to the Swedish police department of about 1 billion pounds ($1.6B).

As we have discussed, there can be an economic case for rationalization. It reduces support costs and increases vendor leverage. However, 1 billion pounds would have bought a lot of support & vendor leverage for the “nonstandard” technologies. It is hard to see that this was an economically rational decision, especially given the particular risks of police work (the system in question is used by officers in the field, to keep records on criminal cases). It does not take much imagination to think of scenarios where a difficult to use system for a police officer could have costly or even tragic results. In this case, the quest for rationalization appears misguided.

A large U.S. company with a robust architecture program found itself with a serious controversy. A request had been made of the Technology Lifecycle Management process for the Subversion version control system to be approved for use by development teams. A central version control team (what might now be called a Release Engineering team) was opposed to this, citing duplication of technologies with their chosen platform of PVCS.

Multiple version management systems already did exist, as different platforms had different requirements. In general, the thinking was that the approval process should simply verify that the tool was “fit for use” and not worry about redundancy.

However, as the discussion escalated, it became clear that the central release engineering team was worried not so much about the technical capabilities of Subversion, but rather a “loss of franchise” for their services — they had been established as the primary shared service for source control. Many teams did use their shared services, but there was also a regulatory issue — they were seen as an authoritative and auditable repository by regulators.

So the situation was complex; was the company “rationalizing” at the technology or service level? Ultimately, a solution (unsatisfactory to most) was arrived at that the Subversion-using team still had to use the central source repository as a system of record. They could develop locally using Subversion but were still expected to archive their source in the central tool.

In the next chapter section, we will go into a more thorough discussion of the service/application “level” versus the technology “level.”

From top to bottom, the data architecture (“what”) perspective includes concepts such as :

We covered data architecture in depth in Chapter 11.

Process architecture is concerned with why and how activities are performed. It includes the detailed, step by step understanding of activities, in a transparent way. From top to bottom, it includes concepts such as:

Importantly, processes CROSS organizations. An “onboard employee” process, as we have seen, may require participation from multiple organizations. We covered process management in depth in Chapter 9.^[15]

The last column in the figure Simplified view of the ArchiMate framework represents steady state activities. “Hire employee” is a process; “Manage Human Resources” is a capability. We do not necessarily know all the steps or details; we just know that if we ask the function or capability for some result, it can produce it. This perspective includes:

We defined function previously in Chapter 9. Note that there is little consensus (and as of 2016 much industry debate) around whether functions are the same as capabilities; this textbook sees them as at least similar. Capability is an important concept in Business Architecture, as it has emerged as the preferred concept for investment. We do not invest in data, or process, except as they are realized by a supporting capability. A comprehensive graphical depiction of “capabilities” may be used to help visualize portfolio investments, sometimes using green/yellow/red color coding — this is called "capability heat mapping.”

Business architecture is defined by the Business Architecture Body of Knowledge (BIZBOK) as “a blueprint of the enterprise that provides a common understanding of the organization and is used to align strategic objectives and tactical demands.”

BIZBOK goes on to say that “the value of Business Architecture is to provide an abstract representation of an enterprise and the business ecosystem in which it operates. By doing so, Business Architecture delivers value as an effective communication and analytical framework for translating strategy into actionable initiatives. The framework also enhances the enterprise’s capacity to enact transformational change, navigate complexity, reduce risk, make more informed decisions, align diverse stakeholders to a shared vision of the future, and leverage technology more effectively.”

BIZBOK covers the Osterwalder Business Model Canvas extensively. [47 pp. 282-297]. In so doing, it clearly implies that the concept of the business model is of interest for business architects. Because of this, it’s helpful to view business architecture as the component of Enterprise Architecture most concerned with the business model, in addition to the operating model.

More specifically, there are a number of concerns that Business Architecture includes:

from [47 p. 2]. The reader might notice some overlap with CObIT enablers, which also include Information, Policies, and Organization.

On the other hand, we DO NOT expect to see in Business Architecture the following:

Application, or application systems, like data, process, and capability, is a fundamental and widely used architecture perspective, as well as a layer. It can be defined as "a fixed-form combination of computing processes and data structures that support a specific business purpose.” [25 p. 125]. An application system is practically relevant, obtainable and operable. (You can buy, or realistically build, one of these).

Application architecture can have two meanings:

For this book, we’ll leave the architecture of a given application for solutions and software architecture. Application architecture is the interaction of multiple applications (which may include digital products and/or services, depending on organization terminology). In a complex, multi-product environment, application architecture tends to focus on the interfaces and interactions between the application systems. It’s often a concern when systems are considered for retirement or replacement (for example, when a comprehensive ERP solution is brought in to replace several dozen home-grown applications).

Application architecture is also concerned with the application lifecycle, as covered at the start of this section.

Where Business Architecture intersects with the business model, technical architecture overlaps with actual engineering and operations. In particular, technical architecture tends to be concerned with:

Solutions architecture, especially, is a loose term. In general, it is restricted to one product, or a few products which are working together, as a “solution” to a business problem. Within that scope, it may incorporate concepts from infrastructure to Business Architecture. It also connotes more technical concerns.

“Architecture” in the sense of pure software is a topic with much research and writing. In this book, it has been a concern since Section 1, and so we don’t talk as much about it here. Software architecture is usually aligned with application architecture, but not all application architecture is software architecture; application architecture may also include packaged solutions whose internal architecture is not a concern.

Information architecture may mean the higher, more business-relevant levels of data architecture. However, the term also is used in relation to application architecture, in the sense of how the user understands the meaning and data represented by a website or application, or even just the navigation structure of a website.

cost of delay is a real and often overlooked issue, in understanding the net value of architecture. But it is only a factor and does not eliminate the value proposition of architecture. If the cost of delay is only a few hundred dollars a month, but the risk or technical debt represent millions, then the delay may be appropriate. Don Reinertsen, who has done more than anyone to promote the idea of cost of delay, emphasizes that all decision-making must take place within an economic framework and that means that the other architectural impact factors on organization value must also be considered [220], chapter 2.

Documentation has been a core concern of the Agile movement, being mentioned in one of the four core principles of the Agile Manifesto:

"Working software over comprehensive documentation” [6+]+.

When documentation primarily takes the form of secondary artifacts, it is appropriate to question the need for it. “The code is the documentation,” some will argue. While it is true that good coding practices result in easier-to-understand (and maintain) source code, the code cannot be the only documentation. As Ruth Malan notes,

…​for systems of sufficient scope and complexity to warrant teams (of teams) working on (incremental) implementation and evolution, the sheer mass of code can make it hard to discover the essential structure from bottom-up decisions made entirely through the medium of code. [177+]+

In terms of systems theory, a complex software system has emergent behavior, not obvious from just looking at its components. Because the system’s behavior can’t be reduced to its pieces, “self-documenting code” can only go so far. The behavior of the assembled components as a system needs to be represented somehow, in a way that transcends the mere mechanics of the pieces. Abstraction is necessary to understand and communicate emergent behavior, and this leads inevitably to visual representation. Without some attention to documenting overall context and systemic intent and behavior, the effectiveness of the overall human/computer system degrades. For example, Alistair Cockburn reports that the Chrysler Comprehensive Compensation project, one of the first widely-reported Agile projects, was eventually halted, and

…​left no archived documentation …​ other than two sentence user stories, the tests, and the code. Eventually, enough people left that the oral tradition and group memory were lost [64 pp. 41-43+]+

In short, failure to sustain a shared mental model of a complex system is a risk that may result in loss of that system’s value.

Agile and DevOps are software-development centric, and have transformed that world. However, digital organizations don’t always build everything. There is a complex web of supplier relationships even for organizations with robust software development capabilities, and many organizations would still prefer to “buy rather than build.” Software may be consuming the world, but that doesn’t mean everyone employs — or should employ — software developers. Agile has not had a primary focus on sourcing, and evaluating commercial software is not a common Agile topic.

Suppose you have an idea for a digital product, and you know that you will be (at least in part) assembling complex services/products produced by others? Suppose further that these provided services overlap (the providers compete)? You need to carefully analyze which services you are going to acquire from which provider. You will need a strategy, and who is it that analyzes these services and their capabilities, interfaces, non-functional characteristics, and makes a final recommendation as to how you are going to bring them all into one unified system?

It is easy to say things like, “the teams get to define their own architecture, ” but at some point, the enterprise must reckon with the cost of an overly diverse supplier base. This is a very old topic in business, not restricted to IT. At the end of the day, supplier and sourcing fragmentation costs real money. Open source, Commercial-off-the-shelf, cloud, in-house…​ the options are bewildering and require experience. In a sense, the supplier base itself is an inventory, subject to aging and spoilage. (We can consider this another way of understanding technical debt). A consistent evaluation approach is important (preferably under an economic framework; see Reinertsen & Hubbard). And at some point, product development teams should not have to do too much of their own R&D on possible platforms for their work.

The Agile Manifesto is well known for saying “The best architectures, requirements, and designs emerge from self-organizing teams” [6]. This is one of the more frequently discussed Agile statements. Former Netflix CTO Adrian Cockcroft has expressed similar views (quote above).

A key question is whether “architecture” is considered at the single product or multi-product level. At the single product level, collaborative teams routinely develop effective software architectures. However, when multiple products are involved, it is hard to see how all the architectural value scenarios are fulfilled without some investment being directed to the goals of cross-product architectural coordination. It helps when rules of the road are established; both Amazon and Netflix have benefited from having certain widely accepted platform standards, such as “every product communicates through APIs.” Netflix has had a long-term commitment to Amazon cloud services; it is probably not acceptable for teams there to decide on a whim to deploy their services on Google Compute Engine or Microsoft Azure, so at least in that sense Netflix has an architecture. The question gets harder when layered products and services with complex lifecycle interactions are involved.

Microservices can reduce the need for cross-team coordination, but as we previously discussed, coordination needs still do emerge.

One principle throughout this book has been “respect the team,” because true product value originates there. If teams are constantly fragmented and their cohesion degraded by enterprise operating models and governance mandates, their ability to creatively solve business problems is hampered. Command and control replace emergence, motivation declines, and valuable creativity is lost. Enterprise architecture must first and foremost protect the precious resource that is the high-performing, collaborative, creative team. As we’ve discussed, imposing multiple governance checkpoints itself adds risk. And while it’s inevitable that the team will be subject to organization-wide mandates, they should be given the benefit of the doubt when autonomy collides with standardization.

When Enterprise Architecture takes on true Business Architecture questions, including how digital capabilities are to be enabled and enhanced, Agile insights become an input or kind of requirement to Business Architecture. What capabilities require high-performing, cross-functional teams? What capabilities can be supported by project-based temporary teams? And what capabilities should be outsourced? The more valuable and difficult the work, the more it calls for the careful development of a common mental model among a close-knit team over time. Driving organizational capability investment into long-running team structures becomes a strategy that organizational architects should consider as they develop the overall organizational portfolio.

Architecture adds value through constraining choices. This may seem counterintuitive, but the choice is often between re-using a known existing platform or engaging in risky research and development of alternatives. R&D costs money, and itself can impose a delay on establishing a reliable digital pipeline. But ultimately, the fundamental objective remains customer and product discovery. All other objectives are secondary; without fulfilling customer needs, architectural consistency is meaningless. Optimizing for the fast creation of product information, tested and validated against operational reality, needs to be top of mind for the architect.

The digital pipeline ultimately is a finely tuned tool for this creation of information. It, itself, has an architecture: business, application, and technical. It operates within an economic framework. To understand the architecture of the digital pipeline is in a sense to understand the “architecture of architecture.”

As we’ve discussed above, architecture, like staff functions generally, is in part a coordination mechanism. It collects and curates knowledge and sustains the organization’s understanding of its complex systems. Architecture also identifies gaps and informs the investment process, in part through collecting feedback from the organization.

If architecture’s fundamental purpose is enabling the right emergent behavior, there are still questions about how it does so. Architecture adds value in assisting when:

As a coordination mechanism, it can operate in various ways including planning, controlling, and collaborating. Each may be appropriate for a given challenge or situation. For example, different approaches are required depending on whether the product challenge is Flower or Cog. A flower is not engineered to fill a gap. A cog is. Market-facing experiments need leeway to pivot, where initiatives intended to fill a gap in a larger system may require more constraints and control. And how do architects know there is a gap? It should be an hypothesis-driven process, that needs to establish that there is a valuable, usable, feasible future state.

As an executing capability, architecture operates in various ways:

Ideally, planning and analysis occur “upstream” of the creation of a product team. In that guise, architecture functions as a sort of zoning or planning authority -— “architecture” is not a process or organization directly experienced by the product team. In this ideal, there is no conflict with product teams because once the team is formed, the architect’s job is done. However, this assumes that all the planning associated with launching a new product or capability was done correctly, and this itself is a kind of waterfall assumption. Some form of feedback and coordination is required in multi-product environments.

It is in the “governance and approval” kind of activity that conflict is most likely to emerge. Cadence and synchronization (e.g.,coordination strategies) with the potential to block teams from pursuing their mission should be very carefully considered. If there is a process or a queue of architecture approvals, it at least should be operated on cost of delay of the work it’s blocking. And more generally, across the organization, the process should be tested against an economic model such as establishing a nominal or portfolio-level cost of delay. Like other processes, architecture itself can be assessed against such a baseline.^[16]

Queued approvals are only one way of solving issues. A rich and under-utilized approach is using internal market-type mechanisms, where overall rules are set, and teams make autonomous decisions based on those rules. Don Reinertsen, in the Principles of Product Development Flow, discusses how Boeing implemented distributed decision-making through setting tradeoff rules for cost and weight. Rather than constantly routing design approvals through a single control point, Boeing instead set the principle that project managers could “purchase” design changes up to $300 per unit, to save a pound of weight. As Reinertsen notes,

The intrinsic elegance of this approach is that the superiors didn’t actually give up control over the decision. Instead, they recognized that they could still control the decision without participating in it. They simply had to control the economic logic of the decision. [220 p. 42].

One particular work product that architects often are concerned with is documentation. The desire for useful documentation, as mentioned above, reflects architecture’s goals of curating a common ground for collaboration. As Bente notes, “In an agile project, explicit care must be taken to ensure proper documentation—for example, by stating it as part of the condition of satisfaction of a user story or in the definition of done” [21 p. 170]

Toyota Kata was discussed in Chapter 7. In Lean Enterprise, Jez Humble et al. argue that it can provide a useful framework for architecture objectives. Toyota Kata emphasizes end-state goals (“target conditions”) and calls for hands-on investigation and response by participating workers, not consultants or distant executives. Architecture can benefit by understanding “gaps” in the sense of Toyota’s target conditions, and then support teams in their collaborative efforts to understand and achieve the desired state. The architectural impact model can assist in thinking through suitable target conditions for architecture:

Keeping the target operating condition specific is preferable. When architecture scopes problems too broadly, the temptation is to undertake large and risky transformation programs. As an alternative, Humble suggests the "strangler pattern,” proposed by Martin Fowler in 2004 [97]. This pattern uses as a metaphor Australian “strangler” vines that grow around trees until the original tree dies, at which point the strangler vine is now itself a sturdy, rooted structure (see Australian strangler vine surrounding tree ^[17]).

To use the strangler pattern is not to replace the system all at once, but rather to do so incrementally, replacing one feature at a time. This may seem more expensive, as it means that both the old and new systems are running (and cost savings through sunsetting the old system will be delayed). But the risk of replacing complex systems is serious and needs to be considered along with any hoped-for cost savings through replacement. Humble and Molesky suggest:

The strangler pattern is proven in practice. Paul Hammant provides a large number of strangler pattern case studies, including:

and others [114].

Of course, there are other ways architecture might add value beyond system replacement, in which case the strangler pattern may not be relevant. In particular, architects may be called on to closely collaborate with product teams when certain kinds of issues emerge. This is not a governance or control interaction; it is instead architecture as a form of shared consulting “bench” or coordination mechanism. Not every product team needs a full-time architect, the reasoning goes, so architects can be assigned to them on a temporary basis, e.g., for one or a few sprints, perhaps of the technical “spike” (discovery/validation/experimentation) variety.

In order to successfully meet this role, the architect needs to have the hands-on technical ability. Many Agile authors are dismissive of “ivory-tower” architects who do not do “hands on” work, and in fact, if an architect is going to sit with a technical team as a solutions advisor they clearly need the technical skills to do so. On the other hand, not all architects operate at the solutions level, nor are the problems they face necessarily programming problems, as you’ll see in the following sidebar, The challenge of the “hands-on” architect.

Finally, how do we evaluate architecture outcomes? If an organization adopts an experimental, Toyota Kata approach, it may find that architecture experiments run on long time horizons. Maintaining an organizational focus on value may be challenging, as the experiments don’t yield results quickly. Curating a common ground of understanding may sound like a fine ideal, but how do we measure it?

First, the concept of Net Promoter Score is relevant for any service organization, internal or external. Its single question “Based on your experience, on a scale of 1-10 would you recommend this product or service to a friend?” efficiently encapsulates value in a single, easy to respond to query.

As digital pipelines become more automated, it may be possible to evaluate their digital exhaust impact on architecture services:

In a world of increasing connectivity and automation, there is no reason for architects in the organization to lack visibility into the consequences of their recommendations. Ultimately, if the cost of operating the coordination mechanism that is architecture exceeds the value, it provides, then continuing to operate it is irrational.