New things in Amazon's Cloud

Amazon AWS made a big announcement yesterday regarding Windows on EC2:
big-day-for-ec2.htm

There are now a number of Windows 2003 server ami options: Amazon Machine Images

Why does any of this matter to GIS markets? GIS distribution has been revolutionized by a battle of the titans Google Map vs Virtual Earth. The popularity of mashups and the continuing spread of location into enterprise business workflow has moved GIS into a browser interface model. However, the backend GIS is still there on servers. Utility cloud computing makes that back end service more affordable to businesses of all sizes, small to large. Even fortune 500 enterprises can make use of auto-scaling load balancing features for ad hoc distribution of location either internally or public facing.

Here are the Amazon Windows AMI offerings:

Amazon Public Images – Windows SQL Server Express + IIS + ASP.NET on Windows Server 2003 R2 (64bit)

Amazon Public Images – Windows SQL Server Express + IIS + ASP.NET on Windows Server 2003 R2 Enterprise Authenticated (64bit)

Amazon Public Images – Windows SQL Server 2005 Standard on Windows Server 2003 R2 Enterprise Authenticated (64bit)

Amazon Public Images – Windows SQL Server Express + IIS + ASP.NET on Windows Server 2003 R2 (32bit)

Amazon Public Images – Windows Server 2003 R2 (32bit)

Amazon Public Images – Windows Server Enterprise 2003 R2 (32bit)

Amazon Public Images – Windows Server 2003 R2 (64bit)

Amazon Public Images – Windows Server 2003 R2 Enterprise (64bit)

Amazon Public Images – Windows SQL Server 2005 Standard on Windows Server 2003 R2 (64bit)

Pricing:

Standard Instances Linux/UNIX Windows
Small (Default) $0.10 per hr $0.125 per hr
Large $0.40 per hr $0.50 per hr
Extra Large $0.80 per hr $1.00 per hr
High CPU Instances Linux/UNIX Windows
Medium $0.20 per hr $0.30 per hr
Extra Large $0.80 per hr $1.20 per hr

Windows prices are only slightly higher than the Linux counterparts and cheaper than GoGrid’s. The small windows instance at Amazon EC2 is $0.125/hr ($3 per day) and includes:

Small Instance (Default) 1.7 GB of memory, 1 EC2 Compute Unit 1 virtual core with 1 EC2 Compute Unit), 160 GB of instance storage, 32-bit platform

A similar GoGrid instance with 2Gb RAM + 160Gb Storage will run 2x$0.19 = $0.38/hr in the "Pay as You Go Pricing", considerably more than the Amazon instance. GoGrid does offer the newer Windows Server 2008 and prepaid plans are less expensive at $0.16 to $0.24 per hr for a similar configuration. Also the slick user interface at GoGrid shows the utility of a visual monitor.

Speaking of user interface, in addition to all of the Windows AMIs there are announcements of future features for EC2:
new-features-for-amazon-ec2

"Management Console – The management console will simplify the process of configuring and operating your applications in the AWS cloud. You’ll be able to get a global picture of your cloud computing environment using a point-and-click web interface."

"Load Balancing – The load balancing service will allow you to balance incoming requests and traffic across multiple EC2 instances. "

"Automatic Scaling – The auto-scaling service will allow you to grow and shrink your usage of EC2 capacity on demand based on application requirements."

"Cloud Monitoring – The cloud monitoring service will provide real time, multi-dimensional monitoring of host resources across any number of EC2 instances, with the ability to aggregate operational metrics across instances, Availability Zones, and time slots."

These will make EC2 easier to use. The Load Balancing and Management Console have been part of GoGrid’s cloud service for awhile now. They do make life easier. Auto-Scaling will be a great help too. Prior to this scaling has been a more or less manual process at EC2. The Windows market is not as used to command line Bash shell scripting so the introduction of visual UI monitor and control makes sense for this new cloud market.

Here is the lowest cost Windows AMI that will be popular with developers:
ami-3934d050
Amazon Public Images – Windows SQL Server Express + IIS + ASP.NET on Windows Server 2003 R2 (32bit)

It includes the basics for ASP .NET 2.0 web apps on IIS 6.0 with SQL Server Express 2005. Of course once on the system it is easy to upgrade to the newer .NET 3.5 and install all of the GIS stack items required.

Here is the procedure I followed for getting my first Windows ami started:

First sign up and pick up a private/public key along with X509 Certificate at EC2.

Download the latest version of ec2-api-tools

Installation includes setting additional environmental variable as described in the ec2 Getting Started Guide.

EC2_HOME
EC2_CERT
EC2_PRIVATE_KEY
EC2_URL

Add to path variable %EC2_HOME%\bin;

After installation verify that the correct ec2-api-tools are installed:
    ec2ver 1.3-26369 2008-08-08

Now we can use the ec2-api-tools:
   ec2-run-instances ami-3934d050 -k gsg-keypair
   ec2-describe-instances <resulting instance id>

Once your instance is running make sure remote desktop service port is open, at least for the default group:
   ec2-authorize default -p 3389

Also you will need to get the randomly assigned administrator password from the new instance using the instance id returned from ec2-describe-instances and the keypair generated earlier:
   ec2-get-password <your instance> -k <full pathname of the gsg_keypair file>

Now it is possible to Remote Desktop to the url furnished by ec2-describe-instances:
   ec2-**-**-**-*.compute-1.amazonaws.com
   User: administrator
   Pass: *********


Fig 1 – Amazon EC2 Windows 2003 basic instance

Summary:
Amazon continues to expand its utility computing cloud. Virtual Windows OS has been a big hole out there, and Linux has grabbed a big lead in that market between Google web compute engines and Amazon EC2. Windows on EC2 opens Amazon utility computing to a much broader segment of the market and pushes deeper into the small business community. The economic turmoil of the times and consequent cost savings imperatives should make utility computing even more attractive to businesses large and small. It remains to be seen if Microsoft’s RedDog announcement at PDC will open a new competitive front in the utility computing world.

A Look at Google Earth API

Google’s earth viewer plugin is available for use in browsers on Windows and Vista, but not yet Linux or Mac. This latest addition to Google’s family of APIs (yet another beta) lets developers make use of the nifty 3D navigation found in the standalone Google Earth. However, as a plugin the developer now has more control using javascript with html or other server side frameworks such as asp .NET. http://code.google.com/apis/earth/
Here is the handy API reference:
http://code.google.com/apis/earth/documentation/reference/index.html

The common pattern looks familiar to Map API users: get a key code for the Google served js library and add some initialization javascript along with a map <div> as seen in this minimalist version.

<html>
  <head>
  <script src=”http://www.google.com/jsapi?key=…keycode…”
  type=”text/javascript”>
  </script>

<script>

var ge = null;
  google.load(“earth”, “1″);

function pageLoad() {
  google.earth.createInstance(“map3d”, initCallback, failureCallback);
  }

function initCallback(object) {
  ge = object;
  ge.getWindow().setVisibility(true);
  var options = ge.getOptions();
  options.setStatusBarVisibility(true);
  ge.getNavigationControl().setVisibility(ge.VISIBILITY_SHOW);
  }

function failureCallback(object) {
  alert(‘load failed’);
  }
  </script>
  <head>
  <body onload=”pageLoad()”>
  <div id=’map3d’></div>
  </body>
</html>
Listing 1 – minimalist Google Earth API in html

This is rather straightforward with the exception of a strange error when closing the browser:

Google explains:
“We’re working on a fix for that bug. You can temporarily disable script debugging in Internet Options –> Advanced Options to keep the messages from showing up.”

…. but the niftiness is in the ability to build your own pages around the api. Instead of using Google Earth links in the standalone version, we can customize a service tailored to very specific desires. For example this experimental Shp Viewer lets a user upload his shape file with a selected EPSG coordinate system and then view it over 3D Google Earth along with some additional useful layers like PLSS from USGS and DRG topo from TerraServer.


Fig 2 – a quarter quad shp file shown over Google Earth along with a USGS topo index layer

In the above example, clicking on a topo icon triggers a ‘click’ event listener, which then goes back to the server to fetch a DRG raster image from TerraServer via a java proxy servlet.

  .
  .
case ‘usgstile’:
  url = server + “/ShpView/usgstile.kml”;
  usgstilenlink = CreateNetLink(“TopoLayer”, url, false);
  ge.getGlobe().getFeatures().appendChild(usgstilenlink);

google.earth.addEventListener(usgstilenlink, “click”, function(event) {
  event.preventDefault();
  var topo = event.getTarget();
  var data = topo.getKml();
  var index = data.substr(data.indexOf(“<td>index</td>”) + 20, 7);
  var usgsurl = server + “/ShpView/servlet/GetDRG?name=” +
  topo.getName() + “&index=” + index;
  loadKml(usgsurl);
  });
  break;
  .
  .
  .
  function loadKml(kmlUrl) {
  google.earth.fetchKml(ge, kmlUrl, function(kmlObject) {
  if (kmlObject) {
  ge.getFeatures().appendChild(kmlObject);
  } else {
  alert(‘Bad KML’);
  }
  });
  }
Listing 2 – addEventListener gives the developer some control of user interaction

In addition to the ‘addEventListener’ the above listing shows how easily a developer can use ‘getKML’ to grab additional attributes out of a topo pologon. Google’s ‘fetchKML’ simplifies life when grabbing kml produced by the server. My first pass on this used XMLHttpRequest directly like this:

  var req;

function loadXMLDoc(url) {
  if (window.XMLHttpRequest) {
  req = new XMLHttpRequest();
  req.onreadystatechange = processReqChange;
  req.open(“GET”, url, true);
  req.send(null);
  // branch for IE6/Windows ActiveX version
  } else if (window.ActiveXObject) {
  req = new ActiveXObject(“Microsoft.XMLHTTP”);
  if (req) {
  req.onreadystatechange = processReqChange;
  req.open(“GET”, url, true);
  req.send();
  }
  }
 }

function processReqChange() {
  if (req.readyState == 4) {
  if (req.status == 200) {
  var topo = ge.parseKml(req.responseText);
  ge.getFeatures().appendChild(topo);
  } else {
  alert(“Problem retrieving the XML data:\n” + req.statusText);
  }
  }
  }
Listing 3 – XMLHttpRequest alternative to ‘fetchKml’

However, I ran into security violations due to running the tomcat Java servlet proxy out of a different port from my web application in IIS. Rather than rewrite the GetDRG servlet in C# as an asmx, I discovered the Google API ‘fetchKML’, which is more succinct anyway, and nicely bypassed the security issue.


Fig 3 – Google Earth api with a TerraServer DRG loaded by clicking on USGS topo polygon

And using the Google 3D navigation


Fig 4 – Google Earth api with a TerraServer DRG using the nifty 3D navigation

Of course in this case 3D is relatively meaningless. Michigan is absurdly flat. A better example of the niftiness of 3D requires moving out west like this view of Havasupai Point draped over the Grand Canyon, or looking at building skp files in major metro areas as in the EPA viewer.


Fig 5 – Google Earth api with a TerraServer DRG Havasupai Point in more meaningful 3D view

This Shp Viewer experiment makes use of a couple other interesting technologies. The .shp file set (.shp, .dbf, .shx) is uploaded to the server where it is then loaded into PostGIS and added to the geoserver catalog. Once in the catalog the features are available to kml reflector for pretty much automatic use in Google Earth api. As a nice side effect the features can be exported to all of geoserver’s output formats pdf, svg, RSS, OpenLayers, png, geotiff, but minus the inimitable background terrain of Google Earth.

ogr2ogr is useful for loading the furnished .shp files into PostGIS where it is accessible to Geoserver and the also very useful kml_reflector. PostGIS + Geoserver + Google Earth API is a very powerful stack for viewing map data. There is also a nice trick for adding Geoserver featureTypes to the catalog programmatically. I used the Java version after writing out the featureType info.xml to geoserver’s data directory. Eventually, work on the Restful configuration version of Geoserver may make this kind of catalog reloading easier.

Summary:
Wow I like 3D and I especially like 3D I have control over. Google Earth API is easy to add into an html page and event listeners give developers some control over features added to the 3D scene. Coupled with the usual suspects in the GIS stack, PostgreSQL /PostGIS and Geoserver, it is relatively easy to make very useful applications. I understand though that real control freaks will want to look at WWjava.

If anyone would like to see a prototype UI of their data using this approach please email

A Noob in Oracle Land

The announcement that Oracle is supporting AMIs in the Amazon cloud came as a surprise to me. I had heard that there was a teaser version of Oracle out there for developers, but had not expected Oracle to jump on the cloud side, especially after Larry Ellison’s recent diatribe against cloud computing.

   “It’s complete gibberish. It’s insane. When is this idiocy going to stop?”

Just curious about this oracle of gibberish, I went on a tour of Oracle Land, the Kingdom of Ellison. This is no small undertaking for an enterprise as ambitious as Oracle. There are endless products and sub-products. The base of the pyramid is the database server, but after buying 50 or more companies in the last year or so, the borders of the empire extend way beyond RDBMS.

The venerable RDBMS has come a long way since IBMs E.F. Codd introduced the concept back in the 70s. I vaguely remember Oracle breaking into the PC world shortly after Turbo Pascal. There was a single DB product for the DOS IBM PC, and documentation consisted of a couple of grayish paperback manuals. Shortly after this, late 80s, a small vendor introduced GeoSQL to hook AutoCAD to the GIS world through Oracle. This was my first introduction to the potential of spatial databases and Oracle. The empire of Ellison has grown since then, and now documentation would fill a library as well as Ellisons bank account.

As an aside, we live in an interesting age at the dusk of the great technology innovators. The infamous industrialists of the previous era now exist only as shadowy figures in history texts, but the business innovators of technology are still walking among us, Larry Ellison, Bill Gates, Steve Jobs. The multi-billion personal fortunes are just now entering the charitable fund phase where our grand children will know their names in some impersonally institutional mode such as the Gates Foundation.

First stop in Oracle Land was a download of the free, as in free beer, teaser version, OracleXE.

  • Total data stored in XE is limited to 4GB
  • XE is limited to 1GB of RAM
  • XE is limited to 1 processor

Since my entire interest in Oracle is the spatial side, my next stop was Justin Lokitz’s helpful article on integration with Geoserver. Leading to this:


Fig 1 -http://localhost:80/geoserver/wms?service=WMS&request=GetMap&format=
image/png&width=800&height=600&srs=EPSG:4326&layers=topp:COUNTIES
&styles=countypopdensity&bbox=-177.1,13.71,-61.48,76.63

Fig 2 – http://localhost:80/geoserver/wms/kml_reflect?layers=COUNTIES

Not a bad start. The Geoserver layer abstracts away the spatial guts of OracleXE. However, curiosity leads on. I found that OracleXE has some spatial components labelled ‘Locator’ as opposed to ‘Spatial’. Though only a subset of the extensive enterprise spatial version, geometry queries are possible. It took me a bit to find my way around.

Interestingly the open source world is generally more helpful in this respect. Although extensive, the forums of commercial software vendors are less friendly. For instance Paul Ramsey of Refraction fame is regularly present on the PostGIS forums, and Frank Warmerdam is always available to give a helping hand at the immensely useful www.gdal.org. But I doubt that I will ever run across a Larry Ellison post on the OracleXE forum. Many posts to commercial forums appear to languish unanswered, which is seldom the case in the OpenSource project forums I monitor.

It is worth noting that gdal’s ogr2ogr can be built with Oracle support on systems with Oracle Client libraries installed.

Oracle’s SDO_Geometry is present in a useful form letting users run geographic join queries like this:

   select c.COUNTY, c.STATE_ABRV, c.TOTPOP, c.POPPSQMI from states s, counties c where s.state = ‘California’ and sdo_anyinteract (c.geom, s.geom) = ‘TRUE’;

My next step was to look at SDO_Geometry in JDBC. Unfortunately Oracle’s JGeometry spatial library is not available for OracleXE, but the LGPL open source JTS library provides helpful OraReader and OraWriter classes. These encapsulate the SDO_GEOMETRY Struct translation to/from jts.geom.Geometry, where the rest of the JTS api can be applied.

logger.info(rsmd.getColumnName(i)+": "+rsmd.getColumnType(i));
st = (oracle.sql.STRUCT) rs.getObject(1);
//convert STRUCT into JGeometry not available in OracleXE
//JGeometry j_geom = JGeometry.load(st);

//JTS to the rescue
OraReader reader = new OraReader();
Geometry geom = reader.read(st);
Coordinate[] coords = geom.getCoordinates();
		.
		.

Next stop, Amazon AWS EC2. Here is a list of the public Oracle AMIs offered::
   Oracle Database 11g Release 1 Enterprise Edition – 64 Bit
   Oracle Database 11g Release 1 Enterprise Edition – 32 Bit
   Oracle Database 11g Release 1 Standard Edition/Standard Edition One – 32 Bit
   Oracle Database 10g Release 2 Express Edition – 32 Bit

The last in the list, OracleXE edition, is the one to experiment with, unless you have a spare Oracle license floating around.

Time to try it:
  C:\>ec2-run-instances ami-7acb2f13 -k gsg-keypair
  C:\>ec2-describe-instances i-??????

and login:

Use of this machine requires acceptance of
the following license agreements.
 1. Oracle Enterprise Linux

http://edelivery.oracle.com/EPD/LinuxLicense/get_form?ARU_LANG=US

 2. Oracle Technology Developer License Terms

http://www.oracle.com/technology/software/popup-license/standard-license.html

 Please enter the above URLs into your browser and review them.
To accept the agreements, enter 'y', otherwise enter 'n'.
Do you accept? [y/n]: y
Thank you.

You may now use this machine.
Welcome to Oracle Database on EC2!
This is the first time this EC2 instance has been started.

Please set the oracle operating system password.
	.
	.
Please specify the passwords for the following database administrative accounts:

SYS (Database Administrative Account) Password:
	.
	.

Now for the link to Apex on the new OracleXE instance:
  http://ec2-??-??-???-??.compute-1.amazonaws.com:8080/apex


Fig 3 – Oracle Apex running from an EC2 OracleXE instance

Looks like we have it.

Summary:
Oracle is the Big Daddy of spatial GIS. It is also the “Mother of all DBA complexity.” Running a spatial app with oracle in the background is not trivial, but it is getting easier. The EC2 OracleXE AMI makes starting an Oracle server instance a matter of minutes. Although lacking some of the capability of its free and open source competition, OracleXE can be useful for the garden variety web enabled spatial app. For the developer with lots of experience in Oracle, OracleXE provides a low cost entry onto the performance/price escalator.

Next on the agenda is adding SDO_GEOMETRY data along with some kind of real spatial rendering, which means in my case getting a tomcat server running with Geoserver on the same OracleXE instance. Alternatively it might be worth a try at installing the OracleXE .rpm on an AMI with a GIS stack already available. And, it will be useful to recompile ogr with oracle db support.

Of course the real mix and match challenge will be OracleXE on an EC2 (real soon now) Windows instance with Java, Tomcat, Geoserver serving a Google Map control coupled to Google Earth, OpenLayers, VirtualEarth. But really EC2 Windows will probably come preconfigured with the new MS SQL Server 2008 and all the promised geospatial goodies including Linq potential.

After just a short trip into the Ellison Empire, I must admit I still like the no frills PostgreSQL/PostGIS better.

AWS to offer Windows + SQL Server


versus

Amazon AWS team just sent out the announcement on a Windows OS offering for later this fall. This confirms rumors floating around on the AWS Roadmap, and will be a significant boost to the EC2 cloud computing. More here … http://aws.typepad.com/aws/2008/10/coming-soon-ama.html

GoGrid has been offering Windows + SQL Server virtual systems for awhile now. It will be interesting to see price comparisons. I imagine that like GoGrid the AWS Windows will cost more because of the MS license issue. The advantage of GoGrid has also been ease of use, hardware balancing, and nice monitoring tools. On the Amazon side is persistent storage S3 and EBS along with SQS. I’m looking forward to trying it out.

Cloud computing is growing. It is important as a platform for GIS. OGC WPS, WMS, WCS, WMS are making a mark on mapping SAAS and cloud platforms fit this model very well.

Posted in AWS