IT managers are looking to computer clusters to optimize Big Data, but they are quickly discovering that clusters, unlike single-purpose servers, are extremely difficult to manage at scale.
Every organization that operates a cluster with traditional data center tools will experience a kind of "Pain Curve" (see diagram). The scale at which each company experiences pain is variable, as it remains a function the company’s size and the quality of its operations staff. Invariably, however, real pain becomes unavoidable for clusters that lack an automated management solution. Automation is the central tool needed to address the requirements for uniform software stacks, consistent service configurations, and total server awareness.
The Exponential Complexity of Managing a Cluster
Due of the rise of Hadoop and OpenStack, many enterprises are now deploying their first small clusters of 10 to 20 servers. At this small scale, the complexity of operating the cluster behaves like 10-20 general data center servers. This is the case on the far left side of the operational complexity graph, before they cross the pain threshold. It is not until the clusters scale – as they inevitably do – when the pain caused by the exponential complexity begins to degrade the cluster’s benefits.
The worldview of a server in a cluster is vastly different from a single-purpose server. By definition, each server in a cluster must be aware of every other server in the cluster. Clustered servers need to accept external requests as a group, process those requests as a group, and then respond to the requests as a group.
At the absolute minimum, each server in a cluster must know about all the other servers in its cluster. Additionally, each cluster service (e.g., Hadoop services) must be configured with the awareness of the other services on all the cluster servers. And, more often than not, each service must be executed on top of the exact same software stack on each cluster server in order to produce consistent results.
In short, all cluster servers must: 1) have the exact same bits on each server; 2) have the exact same software configuration; and 3) have total awareness of each of the cluster servers. As new servers are added to a cluster, the new servers must satisfy all three of these requirements. In addition, the existing clustered servers must now become aware of any newly added servers.
The single-purpose servers in the data center behave much differently. Since each server there operates individually, the complexity added to the operations team increases by a linear amount as new servers are added. It is linear because new servers do not require configuration changes to the existing servers.
Contrast this with the total awareness requirement for clustered servers. Every time a new node is added to the cluster, all the existing servers have to be reconfigured to be made aware of any new servers. The complexity of this task compounds exponentially every time the cluster grows. To show this stark contrast mathematically, the complexity of operations for the general data center servers is O(N), where N = number of servers, whereas the complexity of operating clusters is O(N2).
Cluster Management in the Real World of Business
Consider an actual example experienced by a leading financial services firm. The company’s operations team had planned a Hadoop cluster, and their projected production cluster was scoped at 100 servers. They had a lot of experience running 100-server clusters, so they believed the situation was under control. As in the past, they built a home-grown software project to manage their small cluster.
However, once the 100-server cluster went into production, the machine generated so much value for the business that demand soon skyrocketed, and they had to scale the cluster up even further. The cluster was expanded to 350 servers, but the homegrown system soon crossed the pain threshold and quickly failed. All 350 servers went down and the cluster effectively became a multi-million dollar deadweight.
Finally, after months of pain, this major company implemented an automated solution that was specifically designed to manage clusters at scale. In just 36 hours, the company was able to put all 350 servers back online again with a sustainably configured architecture.
Taking Off the Blinders With Cluster Automation
Why did a global financial services firm have such trouble solving this seemingly straightforward problem? Because the view of a single-purpose server in a traditional data center is entirely different from a cluster. A single-purpose server only accepts external requests, processes those requests, then responds to the requester.
Think of traditional servers like thoroughbred horses that wear blinders over their eyes to avoid distractions during a race. Such a detached server has no awareness of any of the other hundreds or thousands of servers that are churning through requests in the adjoining racks.
As new servers are added to the data center, they are racked and stacked, installed and configured, then brought online -- but the existing servers remain untouched. This level of coordination is what makes clusters the obvious choice for next-gen Big Data platforms like Hadoop, and cloud architectures like OpenStack.
Clusters deliver vastly greater speed, power, and agility. But as the financial services giant found out the hard way in our example, making the shift to clusters becomes too complex without a predesigned software solution. IT teams can only leverage the full value of Big Data analytics by implementing an automated management system to run their clusters.
Greg Bruno is the o-founder and Vice President of Engineering at StackIQ.