While doing some research for an upcoming article, some thoughts gelled about where Agile Software Development lies on its evolutionary path. I was procrastinating somewhat, and while poking around some aviation-related sites I saw an image of a jet engine used on the current crop of regional jets.
Today's jet engine is a marvel of efficiency and has become absolutely mundane - we just expect it to work without so much as a second thought. These engines are also a blend of both simplicity and complexity - simplicity in their design and operation, but complexity in the engineering and materials required to achieve the levels of efficiency and reliability we now take for granted.
If we rewind the clock to 1945 and the end of WWII, jet engines were neither ubiquitous nor reliable. Piston engine powerplants had reached the pinnacle of their evolution with examples such as the Pratt amp; Whitney R-4360 or the Bristol Centaurus. These engines were marvels in their own right, developing enormous amounts of horsepower for their weight - 1.11 hp/lb in the case of the R-4360.
The best jet engines of that era, though, generated relatively little thrust compare to today's engines - the Rolls-Royce Derwent produced about 2,400 lbs. of static thrust. Those engines used considerable amounts of fuel, and were not terribly reliable. The service life of a 1945 vintage jet engine was about 25-50 flying hours. Despite these shortcomings, the new jets showed tremendous promise. Even the first production engines had a power to weight ratio of over 2 to 1, almost twice that of the best piston engines in service at the time. The jets could also propel their aircraft at much higher speeds.
So, after the end of WWII there was little room for improvement for piston engines. They were bulky, heavy, and complex with many moving parts with some pilots and mechanic suggesting that each individual engine has its own personality! Turbine engines in the form of pure jets or turboprops were the future, and they became the focus of virtually all research and development.
Within 5 years of the end of the war, considerable progress had been made in the development of jet propulsion. The deHavilland Comet was flying, powered by 4 Rolls-Royce Avon engines. While the Comet eventually ran into problems related to metal fatigue, the Avon engines were orders of magnitude more reliable than engines just 5 years older. Fuel consumption was lower, and the thrust to weight ratio was now more than 5 to 1.
Improvements and developments have continued since, to the point that today's turbine engines have service lives measured in tens of thousands of flight hours and barely sip fuel compared to their predecessors. These improvements are the result of better engineering practices, enabled in large part by computer aided design and research practices, and better materials used for the engines' construction.
So, how does this apply to Agile Software Development? Consider for a moment the adoption rate of Agile, and the continued use of more traditional approaches to software development. We already know that Agile has a higher "thrust to weight ratio" than traditional processes, but it's still lacking somewhat in service life.
Applying the evolutionary scale of jet engine technology to software development processes, we're probably around the 1950 to 1954 mark. The deHavilland Comet utilized jet engines to become the new standard in speed for large-scale passenger transportation. Tragically, though, that aircraft suffered several catastrophic failures owing to the previously unknown factor of metal fatigue induced by repeated pressurization-depressurization cycles. While there were piston-engined aircraft that were pressurized, it was the jet engines of the Comet that allowed the aircraft to fly higher and faster, thus exposing it to the fatigue problem more often.
The current state of Agile Software Development is similar. We have seen stories of Agile adoptions that have failed. I personally have experienced