CAELinux on Amazon Elastic Cloud Computing infrastructure
Do you wish to run a large simulation job but you cannot afford to pay for an high-end server with 64gb of RAM? You CFD simulation are taking days on your laptop and you would like to speed-it up using an "on-demand" Cluster?
CAELinux in the Cloud may be the solution you were waiting for !
Using Amazon Elastic Compute Cloud, you can now run as many CAELinux servers, on-demand, with the hardware that fits your simulation needs (up to 8 cores / 64 GB RAM) and this just by paying the server hours !
Update: for an up to date procedure please follow this thread on the forum
Description
Amazon Elastic Cloud Computing (EC2, see http://aws.amazon.com/ec2/ ) is the one of the largest virtual machine hosting service and allows to start "on-demand" as many servers as one may require to achieve a given computational task. The service is available to anyone and the execution of the server instance is charged per hour at a very reasonnable price. The main advantage of EC2 for scientific computation is that it allows to run extremelly large jobs on huge servers having up to 64Gb of RAM and 8 cores for a few hours without the need to buy an expensive high-end server hardware.
To benefit from this great flexibility, CAELinux 2010 has just been transfered and adapted to run on Amazon EC2 cloud, so that anyone can run the image on the EC2 server farms on-demand with all the great CFD/FEA softwares pre-installed and ready to run your simulation jobs. Another great advantage of CAELinux in the Cloud is that you do not pay expensive simulation software licences, but just the price of running the hardware, so you can run for example very large CFD jobs spanning for examples 4 nodes with 8 cores each without paying a cent for licences!!
To access the running instance on EC2, you can either use a traditionnal SSH terminal and SFTP to transfer files to / from your workstation. But you can also run CAELinux in the Cloud as a virtual desktop thanks to the pre-installed NX remote desktop server. Using the free NoMachine NX Client on your workstation, you can thus run a full featured Linux CAE workstation remotelly through a basic internet connexion.
This brand new CAELinux EC2 image is currently released as BETA version and is provided for testing purpose only. The goal of this 1st release is to allow you to test the service and give us some feedback on what needs to be improved and what you would need.
Using CAELinux in the Cloud in Amazon EC2
First of all you will need a working Amazon Web Service (AWS) account to use the Amazon EC2 service. For more information on Amazon AWS please see http://aws.amazon.com
To run a CAELinux instance:
1) Sign in in the AWS management console at http://aws.amazon.com and enter the EC2 tab,
2) select Region = EU West and move to the "Images AMIs" (Amazon Machine Image) category in the left menu
3) Select the filters "Public images" and search for the keyword "CAELinux".
You should see the following server image:
| AMI ID | Source | Kernel ID | Architecture | Root |
| ami-360b3b42 | 008531201741/CAELinux2010.2-20110730 | aki-2e4a7f5a | x86_64 | EBS |
4) Right click on the image and choose "Launch instance" to run a server using this image.
Choose the number of instance to run (1 is a good guess ;-) )
Choose the type of server hardware (Large, m1.large, 7.5Gb RAM, 2 cores is a good default value)
Please note that the price per hours depends heavily on the server type see http://aws.amazon.com/ec2/#pricing for more info. Use Spot Instances for the lowest price: http://aws.amazon.com/ec2/spot-instances/
Continue the wizard (leave kernel / ramdisk settings on default) and select or create a "key pair" if necessary.
If not done already, create a Firewall / Security group rule to allow SSH access ( tcp port 22, source 0.0.0.0/0).
Validate the option of the Launch Wizard .
5) Go to Instances tab: you should see the status of your instance(s). After around 2-3 minutes, the server shouldbe accessible.
To access a running instance :
A running instance has two ip adresses, one for the internal network inside the AWS cloud and a second one that is accessible from anywhere on the internet. These two ip adresses correspond to two DNS names. The one you need to know to access your server should look like ec2-XX-XX-XXX-XX.eu-west-1.compute.amazonaws.com
Select the running instance and in the lower tab , copy the Public DNS field in the clip board for further use.
a) To access your server using SSH (terminal):
use your favorite SSH client (ssh in linux, putty on windows machines) to connect to your server at the address ec2-XX-XX-XXX-XX.eu-west-1.compute.amazonaws.com (paste the Public DNS address to avoid mistakes). Authentication is based only on your Amazon SSH keypair, so put it in a convenient directory and make sure that you remove read-write permissions for all others: chmod go-rwx mykeypair.pem
then connect using:
ssh -i mykeypair.pem ubuntu@ec2-XX-XX-XXX-XX.eu-west-1.compute.amazonaws.com
once logged in, define a password for the ubuntu user:
passwd
b) then to access your server with NX Client (remote desktop) :
First, install NoMachine NX Client on your machine: see http://www.nomachine.com/download.php
Then setup a new connection using the NX Connection Wizard, specify
host = ec2-XX-XX-XXX-XX.eu-west-1.compute.amazonaws.com (paste the Public DNS address to avoid mistakes)
connection type = ADSL,
choose the type of Desktop = GNOME (instead of KDE)
choose a desktop size (1024x768 is a minimum)
continue the wizard and then log in using
login : ubuntu
password: "Your custom pasword defined in a)"
After 1 minute max (the 1st time), you should have a window with the remote CAELinux desktop running on Amazon EC2.
c) to transfer files to/from your server :
simply use an SFTP or SCP client to connect to your server . In Windows,you can useFileZilla or WinSCP to connect and transfer files.
On Linux, FileZilla is a good SFTP client. You can also directly use the terminal scp commandor use sshfs to mount a remote folder locally.The simplest option in Linux GNOME, you can directly mount an SSH remote folder using the "Connect to Server" tool.
Whatever tool you use, set the host address to ec2-XX-XX-XXX-XX.eu-west-1.compute.amazonaws.com and use the login = ubuntu and password="your custom password" to connect
Data storage:
By default, the data storage coming with the CAELinux image is only kept during the life of the instance. In other words, if you terminate the instance, the data are lost. To keep the data, you can decide to Stop an instance and restart it later.
Once running, you will notice that the disk space provided by the main system disk is limited (about 2.5 Gb of free space).
For temporary & local storage you can benefit from the local disk of the server. This local disk is accessible at /mnt or at /home/ubuntu/localdisk. Code-Aster has been configured to use the local disk for scratch space, but if you run large simulation, make sure that you configure your simulation tools to put temporary files on /mnt.
If you want persistent storage, the simplest thing to do is to create a new volume in Amazon AWS console and to attach it to your running instance. In this way, you will be able to keep your data even after the EC2 instance is rebooted.
To Stop a CAELinux instance:
To Stop a running instance and thus stop paying for the service, go to AWS management console, in the EC2 Tab, Instance section. Right click on the Instance you want to terminate and choose "Terminate" in the menu. Approx 1 minute latter, the machine will be turned off and all its content will be lost (reset to the default image). To keep the changes, choose "Stop" instead of Terminate. Be carefull though, EBS Instance Images are not made for data persistence, use attached EBS Volumes for that purpose.
For more details, read the Amazon EC2 documentation at http://aws.amazon.com/ebs/ for more information on Elastic Block Store (EBS)
IMPORTANT NOTE : IF YOU DON'T WANT TO PAY FOR UNUSED HOURS, YOU MUST TERMINATE YOUR INSTANCE FROM AWS MANAGEMENT CONSOLE !!
IMPORTANT NOTE2: If you TERMINATE an instance, all the data it contains is LOST. To keep the data, choose Stop instead. The EC2 EBS image is not meant to be persistent, you should use external EBS volumes for data persistence in the cloud.
User Feedback and Requests
Please, leave us your feedback and comments on the forum.
Don't forget that it is still a beta version.
Cluster in the Cloud
The present CAELinux in the Cloud image contains all the tools to run multiprocessor simulation jobs but is not preconfigured to run jobs on a cluster of EC2 virtual machines.
But, in a near future, CAELinux in the Cloud will be extended to include a simple application to setup a "CAELinux Cluster In the Cloud" by configuring a set of running CAELinux EC2 instances to run large MPI jobs. This will allow you to run MPI jobs on your virtual cluster using Code-Aster, Code-Saturne, OpenFOAM or Elmer. To help us maintain the development of CAELinux, this Cluster in the Cloud service will be offered as a paid-for annual subscription based service (but at a very reasonnable price).