Managing Digital: Concepts and Practices

In this section, we will cover:

There is a discussion of business process modeling fundamentals in the appendix.

We’ve defined execution as the point at which supply and demand are combined, and of course, we’ve been executing since the start of our journey. Now, however, we are executing in a more complex environment; we have started to scale along the AKF scaling cube y-axis, and we have either multiple teams working on one product and/or multiple products. Execution becomes more than just “pull another story off the Kanban board.” As multiple teams are formed (see Multiple feature teams, one product), dependencies arise, and we need coordination. The term "architecture” is likely emerging through these discussions. (We will discuss organizational structure directly in Chapter 9, and architecture in Chapter 12).

As noted in the discussion of Amazon’s product strategy, some needs for coordination may be mitigated through the design of the product itself. This is why APIs and microservices are popular architecture styles. If the features and components have well defined protocols for their interaction and clear contracts for matters like performance, development on each team can move forward with some autonomy.

But at scale, complexity is inevitable. What happens when a given business objective requires a coordinated effort across multiple teams? For example, an online e-commerce site might find itself overwhelmed by business success. Upgrading the site to accommodate the new demand might require distinct development work to be performed by multiple teams (see Coordinated initiative across timeframes).

As the quote from Gary Hamel above indicates, a central point of coordination and accountability is advisable. Otherwise, the objective is at risk. (It becomes “someone else’s problem.”) We will return to the investment and organizational aspects of multi-team and multi-product scaling in Chapters 8 and 9. For now, we will focus on dependencies and operational coordination.

What is a "dependency"? We need to think carefully about this. According to the definition above (from [178]), without dependencies, we do not need coordination. (We’ll look at other definitions of coordination in the next two chapters). Diane Strode and her associates [258] have described a comprehensive framework for thinking about dependencies and coordination, including a dependency taxonomy, an inventory of coordination strategies, and an examination of coordination effectiveness criteria.

To understand dependencies, Strode et al. [259] propose the framework shown in Dependency taxonomy (from Strode) footnote:[adapted from [259].

We can see examples of these dependencies throughout digital products. In the next section, we will talk about coordination techniques for managing dependencies.

Our previous discussion of work management was a simple, idealized flow of uniform demand (new product functionality, issues, etc.). Tasks, in general, did not have dependencies, or dependencies were handled through ad hoc coordination within the team. We also assumed that resources (people) were available to perform the tasks; resource contention, while it certainly may have come up, was again handled through ad hoc means. However, as we scale, simple Kanban and visual Andon are no longer sufficient, given the nature of the coordination we now require. We need a more diverse and comprehensive set of techniques.

As we scale up, we see that dependencies and resource management have become defining concerns. However, we retain our Lean Product Development concerns for fast feedback and adaptability, as well as a critical approach to the idea that complex initiatives can be precisely defined and simply executed through open-loop approaches. In this section, we will discuss some of the organizational responses (techniques and tools) that have emerged as proven responses to these emergent issues.

Strode et al also provide a useful framework for understanding coordination mechanisms, excerpted and summarized into Coordination taxonomy (from Strode) footnote:[adapted from [258].

The following sections expand the three strategies (structure, synchronization, boundary spanning) with examples.

Diane Strode and her colleagues propose that coordination effectiveness can be understood as the following taxonomy:

Coordinated execution means that teams have a solid common ground of what they are doing and why, who is doing it, when to do it, and where to go for information. They also have the material outcomes of the right people being in the right place doing the right thing at the right time. These coordination objectives must be achieved with a minimum of waste, and with a speed supporting an OODA loop tighter than the competition’s. Indeed, this is a tall order!

Where project and process management are explicitly coordination-oriented, product management is broader and focused on outcomes. As noted previously, it might use either a project or a process management to achieve its outcomes, or it might not.

Release management was introduced in Part I, and has remained a key concept we’ll return to now. Release management is a common coordination mechanism in product management, even in environments that don’t otherwise emphasize processes or projects. At scale, the concept of a “release train” is seen. The Scaled Agile Framework considers it the “primary value delivery construct” [234].

The train is a cadenced synchronization strategy. It “departs the station” on a reliable schedule. As with Scrum, date and quality are fixed, while the scope is variable. The Scaled Agile Framework (SAFe) emphasizes that “being on the train” in general is a full-time responsibility, so the train is also a temporary organizational or programmatic concept. The release train “engineer” or similar role is an example of the coordinator role seen in the Strode coordination tools and techniques matrix.

The release train is a useful concept for coordinating large, multi-team efforts, and is applicable in environments that have not fully adopted Agile approaches. As author Joanna Rothman notes, “You can de-scope features from a particular train, but you can never allow a train to be late. That helps the project team focus on delivering value and helps your customer become accustomed to taking the product on a more frequent basis” [229].

Project management adds concerns of task ordering and resource management, for efforts typically executed on a one-time basis. Project management groups together a number of helpful coordination tools which is why it is widely used. These tools include:

However, project management also has a number of issues:

(But not all large management activities involve the creation of new information! Consider the previous example of upgrading the RAM in 80,000 POS terminals in 2000 stores).

The project paradigm has a benefit in its explicit limitation of time and money, and the sense of urgency this creates. In general, scope, execution, limited resources, deadlines, and dependencies exist throughout the digital business. A product manager with no understanding of these issues, or tools to deal with them, will likely fail. Product managers should, therefore, be familiar with the basic concepts of project management. However, the way in which project management is implemented, the degree of formality, will vary according to need.

A project manager may still be required, to facilitate discussions, record decisions, and keep the team on track to its stated direction and commitments. Regardless of whether the team considers itself “Agile,” people are sometimes bad at taking notes or being consistent in their usage of tools such as Kanban boards and standups.

It is also useful to have a third party who is knowledgeable about the product and its development, yet has some emotional distance from its success. This can be a difficult balance to strike, but the existence of the role of Scrum coach is indicative of its importance.

We will take another look at project management, as an investment management approach, in Chapter 8.

Approvals are a particular form of activity dependency, and since approvals tend to flow upwards in the organizational hierarchy to busy individuals, they can be a significant source of delay and as Reinertsen points out [219 p. 108], discovering “invisible electric fences” by trial and error is both slow and also reduces human initiative. One boundary spanning coordination artifact an organization can produce as a coordination response is a statement of decision rights, for example, a RACI analysis. RACI stands for

A RACI analysis is often used when accountability must be defined for complex activities. It is used in process management, and also is seen in project management and general organizational structure.

Some agile authors call for an “ECI” matrix, with the “E” standing for empowered, defined as both Accountable and Responsible.

We discussed the emergence of process management in Chapter 5, and in Chapter 6 the basic digital processes of Change, Incident, Problem, and Request management. You should also review the process modeling overview in the appendix.

As we saw in the Strode dependency taxonomy, waiting on a business process is a form of dependency. But business processes are more than just dependency sources and obstacles; they themselves are a form of coordination. In Strode’s terms, they are a boundary spanning activity. It is ironic that a coordination tool itself might be seen as a dependency and blockage to work; this shows at least the risk of assuming that all problems can or should be solved by tightly-specified business processes!

Like project management, process management is concerned with ordering, but less so with the resource load (more on this to come), and more with repeatability and ongoing improvement. The concept of process is often contrasted with that of function or organization. Process management’s goal is to drive repeatable results across organizational boundaries. As we know from our discussion of product management, developing new products is not a particularly repeatable process. The Agile movement arose as a reaction to mis-applied process concepts of "repeatability” in developing software. These concerns remain. However, this book covers more than development. We are interested in the spectrum of digital operations and effort that spans from the unique to the highly repeatable. There is an interesting middle ground, of processes that are at least semi-repeatable. Examples often found in the large digital organization include:

We will discuss a variety of such processes, and the pros and cons of formalizing them, in the Chapter 9 section on industry frameworks. In Chapter 10, we will discuss IT governance in depth. The concept of “control” is critical to IT governance, and processes often play an important role in terms of control.

Just as the traditional IT project is under pressure, there are similar challenges for the traditional IT process. DevOps and continuous delivery are eroding the need for formal change management. Consumerization is challenging traditional internal IT provisioning practices. And self-service help desks are eliminating some traditional support activities. Nevertheless, any rumors of an “end to process” are probably greatly exaggerated. Measurability remains a concern; the Lean philosophy underpinning much Agile thought emphasizes measurement. There will likely always be complex combinations of automated, semi-automated, and manual activity in digital organizations. Some of this activity will be repeatable enough that the “process” construct will be applied to it.

Project management and process management interact in two primary ways as Process and project illustrates:

As Richardson notes in Project Management Theory and Practice, “there are many organizational processes that are needed to optimally support a successful project.” [221] For example, the project may require predictable contractor hiring, or infrastructure provisioning, or security reviews. The same is true for product teams that may not be using a “project” concept to manage their work. To the extent these are managed as repeatable, optimized processes, the risk is reduced. The trouble is when the processes require prioritization of resources to perform them. This can lead to long delays, as the teams performing the process may not have the information to make an informed prioritization decision. Many IT professionals will agree that the relationship between application and infrastructure teams has been contentious for decades because of just this issue. One response has been increasing automation of infrastructure service provisioning (private and external cloud).

Processes can generate various emotional reactions:

“Dysfunctional! Insanity!” (as above)

“Follow the process!”

“What bureaucracy!”

“Don’t create the 'Process Police'!”

“I am an IT Service Management professional. I believe in the ITIL framework!”

“I don’t write code on an assembly line!”

Such reactions are commonplace in social media (and even well-regarded professional books), but we need a more objective and rational approach to understand the pros and cons of processes. We have seen a number of neutral concepts towards this end from authors such as Don Reinertsen and Diane Strode:

A process is a technique, a tool, and no technique should be implemented without a thorough understanding of the organizational context. Nor should any technique be implemented without rigorous, disciplined follow-up as to its real effects, both direct and indirect. Many of the issues with process come from a cultural failure to seek an understanding of the organization needs in objective terms such as these. We’ll think about this cultural failure more in the Chapter 9 discussion of Toyota Kata.

A skeptical and self-critical, “go and see” approach is, therefore, essential. Too often, processes are instituted in reaction to the last crisis, imposed top down, and rarely evaluated for effectiveness. Allowing affected parties to lead a process re-design is a core Lean principle (kaizen). On the other hand, uncoordinated local control of processes can also have destructive effects as discussed below.

Since our initial discussions in Chapter 5 on Work Management, we find ourselves returning full circle. Despite the various ways in which work is conceived, funded, and formulated, at the end “it’s all just work.” The digital organization must retain a concern for the “human resources” (that is, people) who find themselves at the mercy of:

The Lean movement manages through minimizing waste and over-processing. This means both removing un-necessary steps from processes, AND eliminating un-necessary processes completely when required. Correspondingly, the processes that remain should have high levels of visibility. They should be taken with the utmost seriousness, and their status should be central to most people’s awareness. (This is the purpose of Andon.

From workflow tools to collaboration and digital exhaust. One reason process tends to generate friction and be unpopular is the poor usability of workflow tools. Older tools tend to present myriads of data fields to the user and expect a high degree of training. Each state change in the process is supposed to be logged and tracked by having someone sign in to the tool and update status manually.

By contrast, modern workflow approaches take full advantage of mobile platforms and integration with technology like chat rooms and ChatOps. Mobile development imposes higher standards for user experience (UX) design, which makes tracking workflow somewhat easier. Integrated software pipelines that integrate application lifecycle management and/or project management with source control and build management are increasingly gaining favor. For example:

See also the previous discussion of ChatOps, which similarly combines communication and execution in a low-friction manner, while providing rich digital exhaust as an audit trail.

In general, the idea is that we can understand digital processes not through painful manual status updates, but rather through their digital exhaust — the data byproducts of people performing the value-add day-to-day work, at speed and with the flow instead of constant delays for approvals and status updates.

One of the most important reasons for repeatable processes is so that they can be measured and understood. Repeatable processes are measured in terms of:

at the most general level, and of course, all of those measurements must be defined much more specifically depending on the process. Operations (often in the form of business processes) generate data, and data can be aggregated and reported on. Such reporting serves as a form of feedback for management and even governance. Examples of metrics might include:

Measurement is an essential aspect of process management but must be carefully designed. Measuring process can have unforeseen results. Process participants will behave according to how the process is measured. If a help desk operator is measured and rated on how many calls they process an hour, the quality of those interactions may suffer. It is critical that any process “key performance indicator” be understood in terms of the highest possible business objectives. Is the objective truly to process as many calls as possible? Or is it to satisfy the customer so they need not turn to other channels to get their answers?

A variety of terms and practices exist in process metrics and measurement, such as:

The Balanced Scorecard is a commonly-seen approach for measuring and managing organizations. First proposed by Kaplan and Norton [148] in the Harvard Business Review, the Balanced Scorecard groups metrics into the following subject areas:

Metrics can be seen as “lower” versus “higher” level. For example, the metrics from a particular product might be aggregated into a hierarchy with the metrics from all products, to provide an overall metric of product success. Some metrics are perceived to be of particular importance for business processes, and thus may be termed Key Performance Indicators. Metrics can indicate past performance (lagging), or predict future performance (leading).

Once processes are measured, the natural desire is to use the measurements to improve them. We discussed Business Process Re-engineering above. In Lean, there are the concepts of kaizen and kaikaku. Kaizen is an incremental process change; kaikaku is a more radical, abrupt shift in the overall process.

There are many ways that process improvement can go wrong.

The solutions are to be found largely within Lean theory.

In the next section, we’ll look at some of the history and theory behind continuous improvement.

Netflix CTO Reed Hastings, in an influential public presentation "Netflix Culture: Freedom and Responsibility," presents a skeptical attitude towards process. In his view, process emerges as a result of an organization’s talent pool becoming diluted with growth, while at the same time its operations become more complex.

Hastings observes that companies that become overly process-focused can reap great rewards as long as their market stays stable. However, when markets change, they also can be fatally slow to adapt.

Netflix’s strategy is to focus on hiring the highest-performance employees and keeping processes minimal. They admit that their industry (minimally-regulated, creative, non-life-critical) is well suited to this approach [120].

The history of continuous improvement is intertwined with the history of 20th century business itself. Before the Industrial Revolution, goods and services were produced primarily by local farmers, artisans, and merchants. Techniques were jealously guarded, not shared. A given blacksmith might have two or three workers, who might all forge a pan or a sword in a different way. The term “productivity” itself was unknown.

Then the Industrial Revolution happened.

As steam and electric power increased the productivity of industry, requiring greater sums of capital to fund, a search for improvements began. Blacksmith shops (and other craft producers such as grain millers and weavers) began to consolidate into larger organizations, and technology became more complex and dangerous. It started to become clear that allowing each worker to perform the work as they preferred was not feasible.

Enter the scientific method. Thinkers such as Frederick Taylor and Frank and Lillian Gilbreth (of Cheaper by the Dozen fame,) started applying careful techniques of measurement and comparison, in search of the “one best way” to dig ditches, forge implements, or assemble vehicles. Organizations became much more specialized and hierarchical, as shown in the accompanying early organization chart by Gilbreth (see Gilbreth “scientific management” organization ^[1]). An entire profession of industrial engineering was established, along with the formal study of business management itself.

Frederick Taylor (1856-1915) was a mechanical engineer and one of the first industrial engineers. In 1911, he wrote Principles of Scientific Management. One of Taylor’s primary contributions to management thinking was a systematic approach to efficiency. To understand this, let’s consider some fundamentals.

Human beings engage in repetitive activities. These activities consume inputs and produce outputs. It is often possible to compare the outputs against the inputs, numerically, and understand how “productive” the process is. For example, suppose you have two factories producing identical kitchen utensils (such as pizza cutters). If one factory can produce 50,000 pizza cutters for $2,000, while the other requires $5,000, the first factory is more productive.

Assume for a moment that the workers are all earning the same across each factory, and that both factories get the same prices on raw materials. There is possibly a “process” problem. The first factory is more efficient than the second; it can produce more, given the same set of inputs. Why?

There are many possible reasons. Perhaps the second factory is poorly laid out, and the work in progress must be moved too many times in order for workers to perform their tasks. Perhaps the workers are using tools that require more manual steps. Understanding the differences between the two factories, and recommending the “best way,” is what Taylor pioneered, and what industrial engineers do to this day.

As Peter Drucker, one of the most influential management thinkers, says of Frederick Taylor:

The history of industrial engineering is often controversial, however. Hard-won skills were analyzed and stripped from traditional craftspeople by industrial engineers with clipboards (see An industrial engineer observing a worker ^[2]), who now would determine the “one best way.” Workers were increasingly treated as disposable. Work was reduced to its smallest components of a repeatable movement, to be performed on the assembly line, hour after hour, day after day until the industrial engineers developed a new assembly line. Taylor was known for his contempt for the workers, and his methods were used to increase work burdens sometimes to inhuman levels. Finally, some kinds of work simply can’t be broken into constituent tasks. High performing, collaborative, problem-solving teams do not use Taylorist principles, in general. Eventually, the term "Taylorism” was coined, and today is often used more as a criticism than a compliment.

The quest for effiency leads to the long-standing management interest in variability and variation. What do we mean by this?

If you expect a process to take 5 days, what do you make of occurrences when it takes 7 days? 4 days? If you expect a manufacturing process to yield 98% usable product, what do you do when it falls to 97%? 92%? In highly repeatable manufacturing processes, statistical techniques can be applied. Analyzing such “variation” has been a part of management for decades, and is an important part of disciplines such as Six Sigma. This is why Six Sigma is of such interest to manufacturing firms.

W. Edwards Deming (1900-1993) is noted for (among many other things) his understanding of variation and organizational responses to it. Understanding variation is one of the major parts of his “System of Profound Knowledge.” He emphasizes the need to distinguish special causes from common causes of variation; special causes are those requiring management attention.

Deming, in particular, was an advocate of the control chart, a technique developed by Walter Shewhart, to understand whether a process was within statistical control (see Process control chart ^[3]).

However, using techniques of this nature makes certain critical assumptions about the nature of the process. Understanding variation and when to manage it requires care. These techniques were defined to understand physical processes that in general follow normal distributions.

For example, let’s say you are working at a large manufacturer, in their IT organization, and you see the metric of "variance from project plan.” The idea is that your actual project time, scope and resources should be the same, or close to, what you planned. In practice, this tends to become a discussion about time, as resources and scope are often fixed.

The assumption is that, for your project tasks, you should be able to estimate to a meaningful degree of accuracy. Your estimates are equally likely to be too low, or too high. Furthermore, it should be somehow possible to improve the accuracy of your estimates. Your annual review depends on this, in fact.

The problem is that neither of these is true. Despite heroic efforts, you cannot improve your estimation. In process control jargon, there are too many causes of variation for “best practices” to emerge. Project tasks remain unpredictable, and the variability does not follow a normal distribution. Very few tasks get finished earlier than you estimated, and there is a long tail to the right, of tasks that take 2x, 3x or 10x longer than estimated.

In general, applying statistical process control to variable, creative product development processes is inappropriate. For software development, Steven Kan states: “Many assumptions that underlie control charts are not being met in software data. Perhaps the most critical one is that data variation is from homogeneous sources of variation.” That is, the causes of variation are knowable and can be addressed. This is in general not true of development work. [146]

Deming (along with Juran) is also known for “continuous improvement” as a cycle, e.g., "Plan/Do/Check/Act” or "Define/Measure/Analyze/Implement/Control.” Such cycles are akin to the scientific method, as they essentially engage in the ongoing development and testing of hypotheses, and the implementation of validated learning. We touch on similar cycles in our discussions of Lean Startup, OODA, and Toyota Kata.

Discussions of efficiency usually focus on productivity that is predicated on a certain set of inputs. Time can be one of those inputs. Everything else being equal, a company that can produce the pizza cutters more quickly is also viewed as more efficient. Customers may pay a premium for early delivery, and may penalize late delivery; such charges typically would be some percentage (say plus or minus 20%) of the final price of the finished goods.

However, the question of time becomes a game-changer in the “process” of new product development. As we have discussed previously, starting with a series of influential articles in the early 1980s, Don Reinertsen developed the idea of cost of delay for product development [219].

Where the cost of a delayed product shipment might be some percentage, the cost of delay for a delayed product could be much more substantial. For example, if a new product launch misses a key trade show where competitors will be presenting similar innovations, the cost to the company might be millions of dollars of lost revenue or more — many times the product development investment.

This is not a question of “efficiency;” of comparing inputs to outputs and looking for a few percentage points improvement. It is more a matter of effectiveness; of the company’s ability to execute on complex knowledge work.

Ken Schwaber, inventor of the Scrum methodology (along with Jeff Sutherland), like many other software engineers in the 1990s, experienced discomfort with the Deming-inspired process control approach promoted by major software contractors at the time. Mainstream software development processes sought to make software development predictable and repeatable in the sense of a defined process.

As Schwaber discusses [240 pp. 24-25] defined processes are completely understood, which is not the case with creative processes. Highly automated industrial processes run predictably, with consistent results. By contrast, complex processes that are not understood require an empirical model.

Empirical process control, in the Scrum sense, relies on frequent inspection and adaptation. After exposure to Dupont process theory experts who clarified the difference between defined and empirical process control, Schwaber went on to develop the influential Scrum methodology. As he notes:

During my visit to DuPont . . . I realized why [software development] was in such trouble and had such a poor reputation. We were wasting our time trying to control our work by thinking we had an assembly line when the only proper control was frequent and first-hand inspection, followed by immediate adjustments. [240 p. 25].

There’s little question that the idea of statistical process control for digital product development is thoroughly discredited. However, this is not only a textbook on digital product development (as a form of R&D). It covers all of traditional IT management, in its new guise of the digitally transformed organization. Development is only part of digital management.

Coordination is a general and abstract topic, and the research from [258, 155] is a good place to start. Practical discussions of coordination are found in [220] (especially Chapter 7-8) and also [219, 254]. From a software practitioner’s perspective see [67]. The history of Scrum is relevant, in particular, [240]. Critiques of statistical process control for software development are found in [30, 217].

The definitive BPM reference is [233]. Good practical applications are found in [44, 116, 245]. Lean theory is also essential, e.g., [286, 227].

Books

Videos

Professional

Research

Other