EC2 Peace of Mind

Elastic Block Store, EBS, is a useful extension of the AWS services offered by Amazon’s cloud platforms. EBS provides a way to add storage in an EC2 instance independent mode. In other words, storage doesn’t have to be tied to the instance storage, but can exist independently as an external volume. The big deal here is the persistence of your data, even if an instance happens to get killed for some reason. An additional backup security is the ability to make snapshots of an EBS volume and store on the S3 service in S3 buckets.

 

The cost is not a great burden:

EBS Volumes

· $0.15 per GB-month of provisioned storage

· $0.10 per 1 million I/O requests

 

 

Amazon EBS Snapshots to Amazon S3

· $0.15 per GB-month of data stored

· $0.01 per 1,000 PUT requests (when saving a snapshot)

· $0.01 per 10,000 GET requests (when loading a snapshot)

 

The basic approach for creating an EBS volume:

1. start an instance

2. create an EBS volume

3. attach the volume to the instance

4. partition and format the volume

5. add data and services to the instance and its attached volume

6. bundle and register the instance as an AMI stored in S3

7. create a snapshot of the EBS volume

 

After this is complete there is a peace of mind knowing that the instance can be reconstructed from backup services.

 

Restore follows this path if the EBS volume is intact:

1. start a new instance from the AMI bundled previously

2. just attach the volume to the new instance

3. repoint the DNS to this new instance server

 

Restore follows this path if the EBS volume is also trashed:

1. start a new instance from the AMI bundled previously

2. create a new volume from the S3 snapshot

3. attach this new volume to the new instance

4. repoint the DNS to this new instance server

 

I have an Open Source GIS stack loaded on a windows ec2 instance and decided it was time to make a conversion to the security of an EBS volume.

The AWS details are here:

http://docs.amazonwebservices.com/AWSEC2/latest/DeveloperGuide/

 

First make sure the latest api_tools are installed – EC2 API version 2008-08-08:

 

Choose the availability zone that matches the zone of the instance you wish to use.

C:\EC2>ec2-create-volume –size 50 –availability-zone us-east-1b

VOLUME vol-******** 50 us-east-1b creating 2008-11-04T15:38:20+0000

 

Once the volume is created it will be noted as “available.”

C:\EC2>ec2-describe-volumes vol-********

VOLUME vol-******** 50 us-east-1b available 2008-11-04T15:38:20+0000

 

Now the volume can be attached to the instance you had in mind.

C:\EC2>ec2-attach-volume vol-******** -i i-******** -d xvdf

ATTACHMENT vol-******** i-******** xvdf attaching 2008-11-04T15:41:48+0000

 

Once the volume is attached, it’s time to ‘remote desktop’ to the windows instance.

Open the Disk Management tool:

Start/administrative tools/computer management /storage Disk Management

 

You should then see the attached EBS volume and be able to add it to the instance with appropriate partition and format.

Partition info:

http://technet.microsoft.com/en-us/library/cc738081.aspx

Partition walk thru:

http://www.bleepingcomputer.com/tutorials/tutorial116.html

 

Once this is done you have an additional drive available referenced to the external EBS volume. In my example the E: drive.

 


Fig 1 – Example of Disk Manager on an EC2 windows instance showing an EBS volume

 

Once you have a useable EBS, how would you go about making it useful to the GIS stack?

In my stack I am using:

PostgreSQL/PostGIS

Tomcat

Geoserver

 

This means I would like to move all of the PostgreSQL data, tomcat webapps, and the geoserver data to the new EBS volume. Then it will be available for snapshot backup.

 

Postgresql data:

Changing Postgresql data to a new location involves a change to the registry. Stop postgresql service, then change registry ImagePath, move the C:\Program Files\PostgreSQL\8.3\data subdirectory to its new EBS location, E:\postgresql_data, and finally restart the service.

Run regedit:

 “HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\pgsql-some version” and on the ImagePath change the –D option to point to a location on your new EBS volume. Here is the wicki entry with details:  Postgresql Wicki for changing PGDATA.

 

Tomcat webapps:

Open %TOMCAT_HOME%/config/server.xml with a text editor. There should be an entry similar to this:

<Host name=”localhost” appBase=”E://tomcat_webapp”

unpackWARs=”true” autoDeploy=”true”

xmlValidation=”false” xmlNamespaceAware=”false”>

Here you can see that I’ve changed the appBase to point to a subdirectory on my E: drive, the EBS volume. Copy the existing webapp subdirectory to this EBS subdirectory.

 

Geoserver data:

Go to the geoserver webapp’s WEB-INF/web.xml and make sure that the GEOSERVER_DATA_DIR points at a location on the EBS volume. Remember to make the change to the web.xml found in the tomcat webapp directory on the new EBS volume. Copy the geoserver data to its new EBS subdirectory.

<context-param>

<param-name>GEOSERVER_DATA_DIR</param-name>

<param-value>E:\geoserver_data</param-value>

</context-param>

 

Now the data for PostgreSQL/PostGIS, Apache Tomcat, and Geoserver will be accruing on an external volume, safe from sudden EC2 instance death. Of course now that EC2 is no longer beta and the SLA agreement is available this should be a rare occurrence.

 

Now to make things even safer lets run a snapshot:

C:\EC2>ec2-create-snapshot vol-********

SNAPSHOT snap-******** vol-******** pending 2008-11-04T22:14:30+0000

 

C:\EC2>ec2-describe-snapshots snap-********

SNAPSHOT snap-******** vol-******** completed 2008-11-04T22:14:30+0000

 

At this point a snapshot of my volume is stored to S3 where I can use it to create a new volume for use in another instance. I can use the snapshot if I’m creating multi instance clusters or if I need to restore my instance.

 

Of course it would also be wise to make an AMI bundle to reflect the changes made to the basic instance, directory pointers regedits etc. Here is the ami bundle guide for windows instances:  AMI Bundle for windows info

You will first need to prepare a bucket on S3 to receive the AMI bundle.  S3 Info

 

 

C:\EC2 >ec2-bundle-instance i-******** -b ec2-windows-bucket -p ec2-windows_image -o <Amazon EC2 Key ID> -w <private access Key>

BUNDLE bun-******** i-******** norm-ec2-windows ec2-windows_image 2008-11-05T15:28:15+0000 2008-11-05T15:28:15+0000 pending

C:\EC2 >ec2-describe-bundle-tasks

BUNDLE bun-******** i-******** ec2-windows ec2-windows_image 2008-11-05T15:28:15+0000 2008-11-05T16:07:08+0000 complete

C:\EC2 >ec2-register ec2-windows/ec2-windows_image.manifest.xml

IMAGE ami-********

 

Summary:

Amazon cloud is now out of beta and comes with independent storage volumes and snapshot capability useful for backup and scaling functions. GIS open source stacks can make use of these options without a huge effort.