PRODUCT DEVELOPMENT AND AGILE METHODS
28 May 2002
Product engineering covers a vast array of products, from automobiles to cell phones, in which hardware components and embedded software are integrated into a product. When automobiles contain hundreds of microprocessors and cell phones' features are controlled by 300,000 lines of embedded code, then software development becomes a key element to new product development for nearly all companies.
By itself, software development can be a frustrating undertaking, but when combined with hardware aspects of product engineering, the problems are magnified. Integrating hardware and software development raises unique organizational and technological problems.
Software Development Methodology
The widespread use of rigorous, plan-driven software engineering methodologies (those characterized by extensive front-end architecture, requirements analysis, and design activities) markedly impedes product engineering agility. However, many product organizations are frustrated with software development and have reacted by imposing these very plan-driven, linear structure, and rigorous processes on their software development process. These are often the very same strategies they have been laboriously "undoing" on the hardware development side (moving toward iterative development, extensive prototyping, cross-functional teams, and concurrent engineering). In some twisted logic, what's good for hardware doesn't seem to be good for software.
Part of the problem is the mantra-like nature of driving down costs by nailing down requirements and design up front. Today, although there are certainly some errors that are the result of front-end process problems, the bulk of variations comes from the external environment. Since these changes cannot be eliminated, the only viable strategy is to drive down the cost of responding to them. Rather than eliminate rework (the traditional objective of process), the new strategy becomes to reduce the cost of rework. We have to combine some modicum of plan-driven development (reducing costs through "just enough" planning) with much improved adaptive development (reducing the cost of change).
Pure plan-driven development assumes that software architecture, requirements, and overall design can be done early and that later stages are mere implementation. Agile development's strategy is different; for example, it strives to reduce the cost of continuous change by developing simple solutions, periodic redesign, and constant testing. Whether this strategy is cheaper than trying to conceptualize and plan everything up front is not germane. If it is impossible to anticipate changes, then the fact that it's cheaper to do so is moot.
Unfortunately, for product companies and others, there is no easy answer to the question of anticipating (planning) or adapting. It is a balancing act driven by risk assessment -- too much anticipatory work may slow progress, lock development into a rigid inflexible direction, and fail to deliver incremental value. Too much adapting work may increase the costs of avoidable oversights and cause delays. In this search for balance, development groups need to differentiate between product architectures (the overall hardware and software combinations and feature allocations to each), software environment architectures (database, operating system, etc.), and software system architectures (class structures, data schemas, modularization, etc.). Each of these requires a different balance of anticipation and adaptation.
Variety of Knoweldge Domains
The range and depth of knowledge required for new product development can be staggering. Microprocessor design requires knowledge about material science engineering, fundamental microelectronics physics, electrical and electronic engineering, and operating systems software -- just as starting points. Bringing engineers from all of these realms together -- engineers in general aren't the most communicative of individuals -- can be perplexing. Project and product managers, in particular, have a daunting task trying to gain knowledge about the problem domain (which may also involve specific knowledge in fundamental sciences -- physics, chemistry, microbiology). "Our project managers spend 95% of their time on electrical and mechanical design," remarked one product development manager. "They view the software people as weird and just leave them alone." The project managers in many product companies come from "hardware" engineering backgrounds, not from "software" engineering. Software development often comes out last on the product manager's comfort list.
Hardware Holds the Software Up, Or, Software Holds the Hardware Up
Hardware and software components never seem to be complete at the same time. Software engineers complain that hardware isn't available, while hardware engineers complain about the opposite. Although software simulations and hardware prototypes can ease the situation for some products, they can be expensive and can oversimplify real-world situations. One company, developing the embedded software for a cell phone, ran into frustrating problems with purchased hardware test equipment from a major vendor. An oil-field service firm finds that simulations can't replicate all the variations of the real world, but "live" testing is prohibitively expensive. Operating system and computer hardware developers are constantly out of phase during development.
At some level, synchronizing hardware and software development cycles will always be challenging and the problems only partially solvable. However, by their selection of a software development approach, companies can either mitigate or exacerbate the problems.
Software Developers Are from Pluto; Everyone Else Is from Mars
As the quote in the knowledge domain section indicates, software engineers are considered weird by "real" engineers. "Real" becomes part of the problem as electrical, mechanical, and material science engineers believe that real engineering can only be derived from math, physics, chemistry, and biology (maybe) and that the terms computer "science" and software "engineering" are misnomers. Since software engineers have a phobia about being labeled somehow inferior, the stage has been set for major ego clashes between hardware and software developers. Throw in a few physicists and the cauldron really gets hot, because physicists -- as everyone knows because they tell us -- are at the top of the scientific heap.
On top of this simmering professional conflict, hardware and software engineers have a poor appreciation for each others' issues. When an electronic engineer working with a circuit board layout defines a "module" with well-defined interfaces, she intends that the interface remain stable -- changes can have major time and cost considerations. Software engineers, particularly in earlier stages of development, consider modules (whose interfaces may be described by APIs) to be more variable.
Another problem with many companies is that although they've established cross-functional hardware teams, software groups continue to be separate. Again, if the development approach is a traditional up-front, plan-driven one, this functional separation seems to make sense: the software group has its requirements, it just needs to go build it. In reality, these separations cause continuous conflicts as crunch time approaches when hardware and software must be integrated.
--Jim Highsmith, Senior Consultant, Cutter Consortium
Product Development and Agile Methods