Scientific Cloud Computing


Managing EC2 Cloud Compute Clusters


The SC2IT is an Interface Toolset that wraps around the Amazon Web Services EC2 API. SC2IT serves two purposes:
  • Configuring EC2 Cloud Instances into a seamless parallel compute cluster
  • Managing and accessing cloud clusters in a simple and intuitive way, using short commands (bash command line) or Java GUIs



The SC2IT bash interface tools

The SC2IT-bash toolset is a collection of scripts that can be used in any Unix-type shell, such as bash or tcsh. SC2IT-bash may work on Cygwin also. SC2IT-bash provides on the one hand quick and intuitive commands for working with EC2 clusters. SC2IT stores configuration information for each active EC2 cluster. Thus the user can simply state:
sc2connect
rather than calling the AWS EC2 API directly:
ssh -i/home/user/.ec2_clust/.ec2_clust_info. 7729.r-de70cdb7/key_pair_user.pem user@ec2-72-44-53-27.compute-1. amazonaws.com.

On the other hand, the SC2IT-bash is a solution for users who intend to log in to the cloud cluster and control it from the command line, as opposed to users who are more comfortable using SC2IT-java functionality from a GUI.

The following functionality is offered:

  • Launch a cluster with n nodes of machine type a
  • Stop and Restart a cluster
  • Add/Remove nodes to/from a cluster
  • Upload files or directories to a cluster
  • Download files or directories from a cluster
  • Execute a command on a cluster
  • List active clusters
  • Describe active clusters
  • Monitor load in a cluster
  • Execute a predefined workflow on a cluster
  • Open a SSH session to work on the cluster as a regular user
  • Open a SSH session to work on the cluster as root

Credentials

Credentials are used to verify that the user has permission to access EC2 resources, and to determine who is billed for usage of the resources. The user must configure an AWS user account before getting started. This involves at the minimum choosing a username, setting up payment information (typically a credit card), and obtaining a pair of "access keys". Further credential files are required to use the SC2IT interface tools. They can be provided in the following ways:

  • Specify the location of keyfiles by setting environment variables in the shell login file (.bashrc/.cshrc)
  • Calling SC2IT commands with the "-a" option to specify a script setting all credential information for that command only

Requirements

There are no requirements beyond a *nix type shell and a valid AWS account.

Technical information

Examples

SC2IT is used for cloud calculations of the MEEP code.

SC2IT works behind the scenes of the w2web browser for the WIEN2k code.

A typical command-line interface workflow could look like this:

// Launch a new cluster with 10 nodes for ABINIT calculations:
$> sc2launch -n 10 -m abinit

// Prepare the calculation:
$> sc2connect -d "mkdir /home/me/ABINITruns/run10"
// Copy input:
$> sc2put abinit_input.tar.gz

// Log in and work:
$> sc2connect
$EC2headnode> cd ABINITruns/run10
$EC2headnode> ... {do stuff} ...
$EC2headnode> exit

// Copy the output files back to the local computer:
$> sc2get ABINITruns/run10

// Terminate the cloud cluster:
$> sc2terminate