Next: Local management-local site
Up: A Model for Cluster
Previous: A Model for Cluster
In this section we will discuss the advantages of cluster
decentralization (or rather, centralization at a department-local level)
in more detail, doing a cost-benefit analysis (CBA) of local
management-local siting, local management-remote siting, and remote
management-remote siting for a variety of typical cluster environments.
The numbers presented in this CBA are a ``best guess'' sort of
approximation and should be refined with actual numbers where available.
It is difficult to discuss cluster computing at any scale in
completely general terms. On the beowulf list, ``your mileage may
vary'' (YMMV) and ``it depends on what you are doing'' are the standard
warning and answer to nearly any complex question. A cluster that works
optimally (in the CBA sense) for one computation won't work at all for a
different computation. For that reason, we need to differentiate
clusters, and cluster problems, at a very early point in the discussion
into two very generic classes:
- Problems (and clusters) that are very sensitive to cluster
architecture and design. Typically these are problems with a relatively
large communication-to-computation ratio, although it might well include
problems with any sort of ``unusual'' bottlenecks or requirements (very
large memory footprint, specialized network, very large or specialized
- Problems (and clusters) that are not particularly sensitive
to the details of cluster architecture and design and that do not
have any special bottlenecks or requirements.
Silly as this distinction may be, it is a crucial one. Problems and
clusters that fit in the former group for all practical purposes must be engineered and operated on a per-problem, per-cluster basis by
the group that uses the cluster. At this point in time the University
simply cannot provide meaningful support for this sort of cluster
computing at the institutional level. As time passes and the cluster
support described in this document is (hopefully successfully)
implemented that may change. At this time, however, it would be a
capital mistake for the University to even consider anything but a local
management model for this sort of cluster.
In is at least possible to describe some fairly ``generic problems''
that fit the latter description, and to describe a ``standard cluster''
architecture that should do just fine to solve them. Remote,
centralized cluster management makes the most sense when the cluster has
a very ``vanilla'' architecture that will work successfully on a wide
range of relatively simple cluster problems. We will therefore focus
most of our attention on problems of this sort.
To make the discussion concrete, let us consider an ``embarrassingly
parallel'' application such as a Monte Carlo computation consisting of
many fully independent sub-computations. We will presume that only a
small amount of data is required to initiate a sub-computation, which
runs for a long time on a single CPU and then returns a small amount of
data that represents the result. Such a computation runs efficiently in
parallel on any number of processors, requires little in the way of
network speed or local storage, and doesn't globally fail if a single
node goes down in the middle of its sub-computation.
In addition, we will consider a more challenging but still fundamentally
simple problem such as a ``coarse grained'' lattice decomposition of
some sort. Each node works on a part of some large space (lattice). To
advance the computation many of the nodes have to communicate results
between nodes before they can proceed, and if a single node goes down in
mid-computation the entire computation dies and must be started over
from the beginning. However, each node still does a lot of
computation for a little bit of communications, and the computation can
thus be scaled up to many nodes with a very generic network
architecture. Also, the computation has no particularly special
requirements in terms of local storage or memory and can easily fit on a
fairly standard node design. However, it does generate a fairly large
set of results, output continuously throughout the computation.
Both of these computations will run efficiently on a very generic
architecture. Let us now analyze the costs of the different ways of
siting the hardware and managing it.
A cluster supercomputer of any design is at heart a client/server LAN.
Some of the costs of installing and managing a LAN scale with the number
of servers. Others are fixed costs that don't scale at all. Still
others scale with the number of clients, or the number of users. As is
the case with any such LAN, primary costs for LAN construction,
maintenance, and administration include items such as:
- Account management - creation, destruction, modification of
fundamental access and groups privileges for all users of the system.
Typically scales with the number of users independent of the number of
clients, sometimes scales with the number of servers as well.
- Disk management - creation of shared server disk resources, their
secure, authenticated exportation to LAN client systems, backup,
retrieval. Scales with number of servers with a very weak dependence on
number of clients.
- Network management - all aspects of managing both clients and
servers on the network. Scales with number of clients plus number of
- System installation - all aspects of installing servers and
clients, depends strongly on operating system. In package-based linux,
small cost that scales with number and kind of packages installed to get
started, then scales with number of servers and (with an independent
scale factor) number of clients.
- Software management should be a nearly fixed cost absorbed mostly
into system installation and thereafter fully automated. Even so, there
is at least a per-package fixed cost for setting up additions,
modifications, updates to a "standard" list of software.
- Security - ensuring the integrity of all data and resource
utilization. A large fixed cost associated with the entire LAN itself,
with per server and per client costs (larger for the servers) and per
user costs. Similar to, and related to, systems management.
- Systems management - monitoring status of all LAN elements,
identifying and fixing problems, reconfiguration, and more. A large
fixed cost associated with the entire LAN itself, with additional per
server and per client costs (larger for the servers). Similar to, and
related to, systems management.
- User support - dealing with the myriad of user problems that
occur, teaching, hand-holding and more. A large variable cost that
scales with the number and competence of the users, the competence of
the systems staff, the quality of the LAN hardware and design, the
number of systems in the LAN, the number of tools in common use in the
LAN and much, much more.
- Hardware support - repairing, replacing, disposing of all
hardware as it ages out, arranging replacements for critical components
in a proactive way, troubleshooting, and so forth. Scales with the
amount of hardware, its quality, the load placed on it by all sources of
hardware stress (users, programs, physical environment).
- Administration - paperwork and job related work of all flavors.
A highly variable cost managed in different ways by different
organizations. Scales at least weakly with number of systems and number
of users both.
These are all services that must be provided and costs that must be paid
for any LAN, including the specialized LANs we call a compute
cluster or beowulf.
In addition, there are certain physical infrastructure costs associated
with a LAN that must be tallied. These are not human or management
costs (detailed above) but are nonetheless far from negligible.
- Power. Clients, servers, and network components are all
electronic and consume electrical power. In very rough numbers it costs
$75 to provide 100 watts of electrical power twenty four hours a
day for one year at $0.08 per kilowatt-hour. In addition, any place
more components are to be located than there is an immediate supply of
electrical power will require remodeling and rewiring to achieve the
required density in supply. This cost scales with number of components
of any given power consumption, or total power consumed.
- Cooling. All the power consumed by any LAN component must be
removed from the environment in a steady state way or it will build up
as heat, damaging components and risking fires. Cooling occurs by many
physical mechanisms in any environment including natural mechanisms, and
the natural mechanisms vary in efficacy with e.g. the outside
temperature, humidity, airflow, and details of the components physical
location. We will assume (again in very rough numbers) that an
electronic component that is consuming 100 watts of electrical power
(all of which is continuously appearing in the immediate environment of
the component as heat) will require roughly 33 watts of power, on
average, to remove that heat. That is, $25 per 100 watt component, per
year. In addition, any place more components are to be located than
there is local cooling capacity will require remodeling achieve the
required capacity. This costs scales roughly with total power consumed
by all components.
- Physical space. It is especially difficult to estimate the cost
of space in a LAN environment. Every workstation location requires at
least desk space for e.g. system unit, monitor, keyboard and mouse in an
office/workspace environment. Servers and cluster nodes require space
that is more typically fully dedicated to computers and provided with
ample power and cooling. In that space, components can reach very high
densities. The cost of the dedicated space may be ``high'' where it
displaces humans or requires extensive remodeling (amortized, of course,
over the lifetime of the space), it may be irrelevant (in new
construction), it may be ``low'' when finding the space is a matter of
cleaning out an unused supply room with plenty of power and cooling
capacity relative to what you plan to put into it. There are additional
nonlinearities in that small spaces may cost more or less, per
component, than big spaces.
- Global network infrastructure. Access to the LAN backbone, and
LAN access to the campus WAN backbone. The former scales roughly with
the number of LAN environments or networked components, the latter is a
fixed cost per LAN.
With these costs in hand at least by name, we are finally in a position
to consider and compare the various location/management schemes.
Next: Local management-local site
Up: A Model for Cluster
Previous: A Model for Cluster
Robert G. Brown