StarCluster is a utility for creating and managing general purpose computing clusters hosted on Amazon's Elastic Compute Cloud (EC2). StarCluster minimizes the administrative overhead associated with obtaining, configuring, and managing a traditional computing cluster used in research labs or for general distributed computing applications. StarCluster utilizes Amazon's EC2 web service to create and destroy clusters of Linux virtual machines on demand.
To get started, the user creates a simple configuration file with their AWS account details and a few cluster preferences (e.g. number of machines, machine type, ssh keypairs, etc). After creating the configuration file and running StarCluster's "start" command, a cluster of Linux machines configured with the Sun Grid Engine queuing system, an NFS-shared /home directory, and OpenMPI with password-less ssh is created and ready to go out-of-the-box. Running StarCluster's "stop" command will shutdown the cluster and stop paying for service. This allows the user to only pay for what they use.
DependenciesStarCluster has a minimal set of dependencies listed below:
- Registered and fully configured Amazon EC2 account.
- Paramiko (python module for ssh, developed against v 1.7.6)
- Boto (python module for aws, developed against v 1.9d)
Software Included in the StarCluster EC2 AMIStarCluster comes with a publically available AMI on EC2 that includes an extremely minimal software stack for distributed/parallel computing. Currently, the AMI is based on Ubuntu 9.04 and comes in both i386 and x86_64 flavors. The AMIs include the following software:
- OpenMPI - Library used for writing/running parallel applications
- Sun Grid Engine - Queuing system for scheduling jobs on the cluster and handling load balancing.
- NFS - Network File System for sharing folders across the cluster.
- SciPy - Scientific algorithms library for Python (compiled against ATLAS for 8-cpu instance types)
- NumPy - Fast array and numerical library for Python (compiled against ATLAS for 8-cpu instance types)
- IPython - An advanced interactive shell for Python.
- and more ...