Alice in Mirrorland – Silverlight 5 Beta and XNA

“In another moment Alice was through the glass, and had jumped lightly down into the Looking-glass room”

Silverlight 5 Beta was released into the wild at MIX 11 a couple of weeks ago. This is a big step for mirror land. Among many new features is the long anticipated 3D capability. Silverlight 5 took the XNA route to 3D instead of the WPF 3D XAML route. XNA is closer to the GPU with the time tested graphics rendering pipeline familiar to Direct3D/OpenGL developers, but not so familiar to XAML developers.

The older WPF 3D XAML aligns better with X3D, the ISO sanctioned XML 3D graphics standard, while XNA aligns with the competing WebGL javascript wrapper for OpenGL. Eventually XML 3D representations also boil down to a rendering pipeline, but the core difference is that XNA is immediate mode while XML 3D is kind of stuck with retained mode. Although you pick up recursive control rendering with XML 3D, you lose out when it comes to moving through a scene in the usual avatar game sense.

From a Silverlight XAML perspective, mirror land is largely a static machine with infrequent events triggered by users. In between events, the machine is silent. XAML’s retained mode graphics lacks a sense of time’s flow. In contrast, enter XNA through Alice’s DrawingSurface, and the machine whirs on and on. Users occasionally throw events into the machine and off it goes in a new direction, but there is no stopping. Frames are clicking by apace.

Thus time enters mirror land in frames per second. Admittedly this is crude relative to our world. Time is measured out in the proximate range of 1/20th to 1/60th a second per frame. Nothing like the cusp of the moment here, and certainly no need for the nuance of Dedekind’s cut. Time may be chunky in mirror land, but with immediate mode XNA it does move, clicking through the present moment one frame at a time.

Once Silverlight 5 is released there will be a continuous XNA API across Microsoft’s entire spectrum: Windows 7 desktops, Windows 7 phones, XBox game consoles, and now the browser. Silverlight 5 and WP7 implementations are a subset of the full XNA game framework available to desktop and XBox developers. Both SL5 and WP7 will soon have merged Silverlight XNA capabilities. For symmetry sake XBox should have Silverlight as apparently announced here. It would be nice for a web browsing XBox TV console.

WP7 developers will need to wait until the future WP7 Mango release before merging XNA and Silverlight into a single app. It’s currently an either/or proposition for the mobile branch of XNA/SL.

At any rate, with SL5 Beta, Silverlight and 3D XNA now coexist. The border lies at the <DrawingSurface> element:

<DrawingSurface Draw="OnDraw" SizeChanged="DrawingSurface_SizeChanged" />

North of the border lies XML and recursive hierarchies, a largely language world populated with “semantics” and “ontologies.” South of the border lies a lush XNA jungle with drums throbbing in the night. Yes, there are tropical white sands by an azure sea, but the heart of darkness presses in on the mind.

XAML touches the academic world. XNA intersects Hollywood. It strikes me as one of those outmoded Freudian landscapes so popular in the 50’s, the raw power of XNA boiling beneath XAML’s super-ego. I might also note there are bugs in paradise, but after all this is beta.

Merging these two worlds causes a bit of schizophrenia. Above is Silverlight XAML with the beauty of recursive hierarchies and below is all XNA with its rendering pipeline plumbing. Alice steps into the DrawingSurface confronting a very different world indeed. No more recursive controls beyond this border. Halt! Only immediate mode allowed. The learning curve south of the border is not insignificant, but beauty awaits.

XNA involves tessellated models, rendering pipelines, vertex shaders, pixel shaders, and a high level shading language, HLSL, accompanied by the usual linear algebra suspects. Anytime you run across register references you know this is getting closer to hardware.

…a cry that was no more than a breath: “The horror! The horror!”

sampler2D CloudSampler : register(s0);
static const float3 AmbientColor = float3(0.5f, 0.75f, 1.0f);
static const float3 LampColor = float3(1.0f, 1.0f, 1.0f);
static const float AmbientIntensity = 0.1f;
static const float DiffuseIntensity = 1.2f;
static const float SpecularIntensity = 0.05f;
static const float SpecularPower = 10.0f;
			.
			.

Here is an overview of the pipeline from Aaron Oneal’s MIX talk:

So now that we have XNA it’s time to take a spin. The best way to get started is to borrow from the experts. Aaron Oneal has been very kind to post some nice samples including a game engine called Babylon written by David Catuhe.

The Silverlight 5 beta version of Babylon uses Silverlight to set some options and SL5 DrawingSurface to host scenes. Using mouse and arrow keys allows the camera/avatar to move through the virtual environment colliding with walls etc. For those wishing to get an idea of what XNA is all about this webcafe model in Babylon is a good start.

The models are apparently produced in AutoCAD 3DS and are probably difficult to build. Perhaps 3D point clouds will someday help, but you can see the potential for navigable high risk complex facility modeling. This model has over 60,000 faces, but I can still walk through exploring the environment without any difficulty and all I’m using is an older NVidia motherboard GPU.

Apparently, SL5 XNA can make a compelling interactive museum, refinery, nuclear facility, or WalMart browser. This is not a stitched pano or photosynth interior, but a full blown 3D model.

You’ve gotta love that late afternoon shadow affect. Notice the camera is evidently held by a vampire. I checked carefully and it casts no shadow!

But what about mapping?

From a mapping perspective the fun begins with this solar wind sample. It features all the necessary models, and shaders for earth, complete with terrain, multi altitude atmosphere clouds, and lighting. It also has examples of basic mouse and arrow key camera control.

Solar Wind Globe
Fig 4 – Solar Wind SL5 XNA sample

This is my starting point. Solar Wind illustrates generating a tessellated sphere model with applied textures for various layers. It even illustrates the use of a normal (bump) map for 3D effects on the surface without needing a tessellated surface terrain model. Especially interesting is the use of bump maps to show a population density image as 3D.

My simple project is to extend this solar wind sample slightly by adding layers from NASA Neo. NASA Neo conveniently publishes 45 categories and 129 layers of a variety of global data collected on a regular basis. The first task is to read the Neo GetCapabilities XML and produce the TreeView control to manage such a wealth of data. The TreeView control comes from the Silverlight Toolkit project. Populating this is a matter of reading through the Layer elements of the returned XML and adding layers to a collection which is then bound to the tree view’s ItemsSource property.

    private void CreateCapabilities111(XDocument document)
    {
        //WMS 1.1.1
        XElement GetMap = document.Element("WMT_MS_Capabilities").Element("Capability")
            .Element("Request").Element("GetMap").Element("DCPType")
            .Element("HTTP").Element("Get").Element("OnlineResource");
        XNamespace xlink = "http://www.w3.org/1999/xlink";
        getMapUrl = GetMap.Attribute(xlink + "href").Value;
        if (getMapUrl.IndexOf("?") != -1) getMapUrl =
                  getMapUrl.Substring(0, getMapUrl.IndexOf("?"));

        ObservableCollection layers = new ObservableCollection();
        foreach (XElement element in
document.Element("WMT_MS_Capabilities").Element("Capability")
                .Element("Layer").Descendants("Layer"))
        {
            if (element.Descendants("Layer").Count() > 0)
            {
                WMSLayer lyr0 = new WMSLayer();
                lyr0.Title = (string)element.Element("Title");
                lyr0.Name = "header";
                foreach (XElement element1 in element.Descendants("Layer"))
                {
                    WMSLayer lyr1 = new WMSLayer();
                    lyr1.Title = (string)element1.Element("Title");
                    lyr1.Name = (string)element1.Element("Name");
                    lyr0.sublayers.Add(lyr1);
                }

                layers.Add(lyr0);
            }
        }
        LayerTree.ItemsSource = layers;
    }

Once the tree is populated, OnSelectedItemChanged events provide the trigger for a GetMap request to NASA Neo returning a new png image. I wrote a proxy WCF service to grab the image and then write it to png even if the source is jpeg. It’s nice to have an alpha channel for some types of visualization.

The difficulty for an XNA novice like myself is understanding the hlsl files and coming to terms with the rendering pipeline. Changing the source image for a Texture2D shader requires dropping the whole model, changing the image source, and finally reloading the scene model and pipeline once again. It sounds like an expensive operation but surprisingly this re-instantiation seems to take less time than receiving the GetMap request from the WMS service. In WPF it was always interesting to put a Video element over the scene model, but I doubt that will work here in XNA.

The result is often a beautiful rendering of the earth displaying real satellite data at a global level.

Some project extensions:

  • I need to revisit lighting which resides in the cloud shader hlsl. Since the original cloud model is not real cloud coverage, it is usually not an asset to NASA Neo data. I will need to replace the cloud pixel image with something benign to take advantage of the proper lighting setup for daytime.
  • Next on the list is exploring collision. WPF 3D provided a convenient RayMeshGeometry3DHitTestResult. In XNA it seems getting a point on the earth to trigger a location event requires some manner of collision or Ray.Intersects(Plane). If that can be worked out the logical next step is grabbing DEM data from USGS for generating ground level terrain models.
  • There is a lot of public LiDAR data out there as well. Thanks to companies like QCoherent, some of it is available as WMS/WFS. So next on the agenda is moving 3D LiDAR online.
  • The bump map approach to displaying variable geographic density as relief is a useful concept. There ought to be lots of global epidemiology data that can be transformed to a color density map for display as a relief bump map.

Lots of ideas, little time or money, but Silverlight 5 will make possible a lot of very interesting web apps.

Helpful links:
Silverlight 5 Beta: http://www.silverlight.net/getstarted/silverlight-5-beta/
Runtime: http://go.microsoft.com/fwlink/?LinkId=213904

Silverlight 5 features:
http://i1.silverlight.net/content/downloads/silverlight_5_beta_features.pdf?cdn_id=1
“Silverlight 5 now has built-in XNA 3D graphics API”

XNA: http://msdn.microsoft.com/en-us/aa937791.aspx
Overview

NASA Neo: http://localhost/NASA-Neo/publish.htm

Babylon Scenes: Michel Rousseau, courtesy of Bewise.fr

Babylon Engine: David Catuhe / Microsoft France / DPE

Summary:

“I am real!” said Alice, and began to cry.

“You won’t make yourself a bit realler by crying,” Tweedledee remarked: “there’s nothing to cry about.”

“If I wasn’t real,” Alice said – half-laughing through her tears, it all seemed so ridiculous – “I shouldn’t be able to cry.”

WMS 3D with Silverlight and WPF


Silverlight view of DRCOG WMS
Fig 1 – Silverlight view of DRCOG WMS Layers

WPF 3D view of DRCOG WMS Layers
Fig 2 – WPF 3D view of DRCOG WMS Layers

Silverlight 2.0 MapControl CTP makes it easy to create a WMS viewer. However, Silverlight exposes only a 2D subset of the full WPF xaml specification. In my experiments so far with LiDAR I can see the various TINS, contours, and point clouds as a 2D view, but what I would really like to do is view data in 3D. Recalling some work I did a few years ago using WPF GeometryModel3D elements I decided to look at a WPF loose xaml approach. This is not restricted to use with LiDAR WMS services. Here in Colorado relief is major part of life and it would be helpful to visualize any number of WMS services using a 3D terrain model. Google and VE(Bing) already do this for us using their proprietary DEM, but eventually I would like to use different DEM resolutions, anticipating Open Terrain sources or in house LiDAR. DRCOG’s WMS service is conveniently available and lies right on the edge of the Rocky Mountain uplift so it is a nice WMS service to highlight relief visualization.

First navigating to the WPF service using Silverlight is a bit different from html. The familiar anchor element is gone. One Silverlight approach is to add a HyperlinkButton:<HyperlinkButton Content=”3D” TargetName=”_blank” NavigateUri=”"/> and change the NavigateUri as needed. Another approach is to use Html.Page.Window.Navigate() or Html.Page.Window.NavigateToBookmark() allowing a click handler to setup the navigate:

        private void ClickTest_Button(object sender, RoutedEventArgs e)
        {
            Uri testwpf = new Uri("http://www.cadmaps.com");
            HtmlPage.Window.Navigate(testwpf, "_blank");
        }

Using one of these Navigate methods the Uri can be pointed at the service used to create the 3D loose xaml.

The next item is getting 3D terrain to populate the xaml mesh. Eventually LiDAR Server will provide 3D surface data, but in the meantime JPL has a nice trick for their srtm and us_ned WMS service. One of the selectable styles is “feet_short_int”. This style will produce a grayscale image with elevation coded into the int16 pixels. Using these elevations I can then create the xaml mesh along with some primitive manipulation tools.

Ref: JPL GetCapabilities

    <Layer queryable="0">
      <Name>us_ned</Name>
      <Title>United States elevation, 30m</Title>
      <Abstract>
        Continental United States elevation, produced from the USGS National Elevation.
        The default style is scaled to 8 bit from the orginal floating point data.
      </Abstract>
      <LatLonBoundingBox minx="-125" miny="24" maxx="-66" maxy="50"/>
      <Style><Name>default</Name> <Title>
Default Elevation</Title> </Style>
      <Style><Name>short_int</Name> <Title>
short int signed elevation values when format is image/png or tiff
</Title> </Style>
      <Style><Name>feet_short_int</Name> <Title>
short int elevation values in feet when format is image/png or image/tiff
</Title> </Style>
      <Style><Name>real</Name> <Title>
DEM real numbers, in floating point format, meters, when used with image/tiff
</Title> </Style>
      <Style><Name>feet_real</Name> <Title>
DEM in real numbers, in floating point format, feet, when used with image/tiff
</Title> </Style>
      <ScaleHint min="20" max="10000" /> <MinScaleDenominator>24000</MinScaleDenominator>
    </Layer>


JPL us_ned 30m
Fig 3 – JPL us_ned 30m can be returned as feet_short_int

My first attempt used a .ashx service handler. I could then obtain the Bitmap like this:

     WebClient client = new WebClient();
     byte[] bytes = client.DownloadData(new Uri(terrainUrl));
     Bitmap bmp = (Bitmap)Bitmap.FromStream(new MemoryStream(bytes), true);

Unfortunately the returned pixel format of the Bitmap is incorrect. Instead of PixelFormat.Format16bppGrayScale the Bitmap returned PixelFormat.Format32bppArgb. This is not exactly helpful, quantizing the elevations. Instead of an elevation at each pixel GetPixel returns an RGB like 2A 2A 2A meaning e = p.B * 256 ^ 2 + p.G * 256 ^ 1 + p.R * 256 ^ 0; is incorrect. It appears that the returned value is quantized on the R value since to be gray each value of RGB must be equal. I checked the image returned by JPL using GdalInfo. This shows 2 band Uint16 values. Since Bitmap reads in a 4 byte value (2 Uint16s) it apparently defaults to Format32bppArgb and is then useless for actual elevation.

GDALINFO
Driver: PNG/Portable Network Graphics
Files: C:\Users\rkgeorge\Pictures\wms.png
Size is 500, 351
Coordinate System is `'
Image Structure Metadata:
  INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left  (    0.0,    0.0)
Lower Left  (    0.0,  351.0)
Upper Right (  500.0,    0.0)
Lower Right (  500.0,  351.0)
Center      (  250.0,  175.5)
Band 1 Block=500x1 Type=UInt16, ColorInterp=Gray
Band 2 Block=500x1 Type=UInt16, ColorInterp=Alpha

color quantized example
Fig 4 – Example of a color quantized elevation from forced 32bppArgb

There is probably some type of pointer approach to pass through the raw bytes correctly. However, services are a nice way to decouple a view from the backend and it is just as easy to use a Java servlet which can take advantage of JAI, so in this case I switched to a Java servlet:

URL u = new URL(terrainUrl);
image = JAI.create(“url”, u);
SampleModel sm = image.getSampleModel();
int nbands = sm.getNumBands();
Raster inputRaster = image.getData();
int[] pixels = new int[nbands*width*height];
inputRaster.getPixels(0,0,width,height,pixels);

The resulting pixels array is int16[] so the correct 30m us_ned elevations are available from the JPL WMS service. Using the resulting elevations I can fill in a Positions attribute of my WPF MeshGeometry3D element. Since this is an array it is also easy to add a TriangleIndices attribute and a TextureCoordinates attribute for the draped image. The final loose xaml will eventually show up in the browser like this:


WPF view of DRCOG layers draped on JPL terrain
Fig 5 - WPF view of DRCOG layers draped on JPL terrain

Because I am simplifying life by using loose xaml there is no code behind for control of the views. However, we can use binding to attach various control values to camera positions and geometry transforms that allow a user to move around the terrain even without a full blown code behind xbap. For example:

<Viewport3D.Camera>
  <PerspectiveCamera
    Position="0 0 " + dimw + ""
    LookDirection="0,0,-1"
    FieldOfView="{Binding ElementName=zoomlensslider, Path=Value}">
    <PerspectiveCamera.Transform>
      <Transform3DGroup>
        <TranslateTransform3D
          OffsetX="{Binding ElementName=hscroll, Path=Value}"
          OffsetY="{Binding ElementName=vscroll, Path=Value}"
        />
      </Transform3DGroup>
    </PerspectiveCamera.Transform>
  </PerspectiveCamera>
</Viewport3D.Camera>

In this case the camera FieldOfView is bound to a slider value and the camera position transform offsetx and offsety are bound to scrollbar values. Binding values to controls in the same xaml file is a powerful concept that provides dynamic capability without any code behind.

Code behind with an xbap would actually make more useful navigation tools possible, but this is a good start. The controls allow the user to zoom the camera lense, change camera x,y position, tilt and rotate the viewport as well as exaggerate the relief scale. Changing to an xbap that handles the terrain image translation with code behind would allow the use of mouse wheel and click hits for pulling up item data fields similar to the Silverlight WMS viewer.

As great as WPF is, it would still be a difficult problem to turn it into a tiled interface like the SIlverlight MapControl. I imagine the good folks at Microsoft are working on it. Basically the mesh elements would be tiled into a pyramid just like imagery tiles. Then these tiles would be loaded as needed. It is easy to imagine an algorithm that would work with the backing pyramid as a 3D view curve. In other words distance from the camera would translate to tiles at higher pyramid levels of lower resolution. A view field would pull tiles from the mesh pyramid using a distance curve so that lower resolution would fill in further ranges and higher resolution at closer ranges. Some kind of 3D will eventually be available but is beyond the scope of my experiment.

Some Problems

1) JPL WMS has some difficulty keeping up with GetMap requests as noted on their website, which means GetMap requests are unreliable. Sometimes there is no issue and other days the load is heavier and there are few times when a GetMap request is possible. Here is their explanation:
"Due to server overloading, client applications are strongly advised to use the existing tile datasets wherever possible, as described in the Tiled WMS or Google Earth KML support

Frequent and repetitive requests for non-cached, small WMS tiles require an excessive amount of server resources and will be blocked in order to preserve server functionality. The OnEarth server is an experimental technology demonstrator and does not have enough resources to support these requests.
An alternative solution already exists in the form of tiled WMS
While sets of tiles with different sizes and alignment can be added when needed, for large datasets the duplication of storage, processing and data management resources is prohibitive."

2) Silverlight works pretty well cross browser at least on my Windows development systems. My tests with Mozilla Firefox managed Silverlight and WPF loose xaml just fine. However, Google Chrome has problems with WPF loose xaml reporting this error:
"This application has failed to start because xpcom.dll was not found"

A quick Google search came up with this:
Chromium does not support XPCOM. It is a Mozilla specific API.
https://developer.mozilla.org/en/XPCOM

More details here:
http://code.google.com/p/chromium/issues/detail?id=4051

3) Another baffling issue arose with the WPF loose xaml. It turns out that each click of the LinkButton or access of the url for the WPF service caused the service to be accessed three times in succession. This was true of both the .ashx and the servlet versions. The User Agent was not distinguishable so I wasn't able to short circuit the first two GET accesses. I only found this true for context.Response.ContentType = "application/xaml+xml"; not text/html etc. So far I haven't seen anything that would solve my problem. It is too bad since the computation for 15Mb WPF files is significant and three times for each result is a bit unnecessary. It may be something quite simple in the response header. I will keep an eye out. There was a similar problem with IE6 years ago, but in that case the service could watch for a Request.UserAgent notation and skip the unecessary duplicate calls.

4) WPF is verbose and 15Mb terrain views take a bit of patience especially considering the above problem. I need to add an auto xaml gzip to my Apache Tomcat. I'm sure there is something similar in IIS if I get around to a pointer solution to the grayscale.

Summary

Silverlight is a natural for interface development and integrates well with WPF ClickOnce for the 3D views that are still not available in Silverlight. My hope is that someday there will be a SIlverlight version capable of 3D scene graph development.

The LiDAR WMS Server roadmap also includes a style for grayscale elevations at some point. That would enable viewing much more detailed high resolution DEM directly from LAS files. I believe this will be a useful advance for use of online LiDAR viewing.

Also a thanks to DRCOG. I'm always glad to see agencies willing to support the wider GIS community by publishing data as OGC services and then further supporting it with adequate server performance. I just wish JPL had a bit more budget to do the same.

Silverlight view of LiDAR Server TIN
Fig 6 - Silverlight view of LiDAR Server TIN

WPF 3D view of LiDAR Server TIN
Fig 7 - WPF 3D view of LiDAR Server TIN

Next Gen UI for GIS

Interesting to think that GIS UI’s are in for some major changes. Next gen GIS users will bring UI expectations from the OGL, DirectX, WPF, X3D, XNA world. It seems the biggest UI transform is just navigating a camera (avatar) in 3D rather than 2.5D.
See VerySpatial’s Lost in the Virtual Fog

GIS UIs currently follow the usual 2.5D OS interactions with primarily Mouse event handling. There are 3D analogies to current 2.5D interaction.

MouseEnter, MouseLeave – rollover
Passive select – 3D intersect media changes (popups, text, film, photo, etc) would correspond to current 2.5D rollover events

MouseClick, Double Click
Active select – corresponding 3D click events are just as necessary in a 3D virtual world – perhaps a virtual laser pointer not much different than the shoot’em game scenario. The virtual point device has a straight line of sight intersect in the scene graph.
WPF 3D hit testing

    RayHitTestParameters hitParams =
        new RayHitTestParameters(
            new Point3D(0, 0, 0),
            new Vector3D(1, 0, 0)
            );
    VisualTreeHelper.HitTest(visual3d, null, ResultCallback, hitParams);

MouseMove, MouseWheel – Pan, Drag, and zoom
Pan and Zoom is analogous to camera control in 3D. Navigation in 3D is a bit more complex including not just 3D camera position, but a 3D vector ‘look direction’ , ‘up direction’, and also ‘field of view’: WPF 3D Tutorial

   <Viewport3D.Camera>
    <PerspectiveCamera
	  FarPlaneDistance="100"
	  LookDirection="-11,-10,-9"
	  UpDirection="0,1,0"
	  NearPlaneDistance="1"
	  Position="11,10,9"
	  FieldOfView="70" />
    </Viewport3D.Camera>

Manipulating a camera in 3D is cumbersome using typical Mouse event handling. 3D navigation replacement of pan and zoom is better suited to a new device altogether. iPhone/WII like inertial sensor navigation can’t be far away. A 3D mouse is just a mouse freed from its 2D pad, a stripped down inertial sensor device or more sensitive WII wand. Now there are direction and tilt events, as well as move events for a more natural camera view control.

Beyond 2.5D analogies
There seem to be some innovative gesture interactions arriving with thanks to the iPhone.


Touch Pack for Windows 7

Extending hand gestures to 3D is further down the road but obviously one step further in the immersive direction. GIS is headed down the virtual reality path.

Stimulus LiDAR


3D VE navigation
Fig 1 – New VE 3D Building – Walk Though Walls

I’ve been thinking about LiDAR recently. I was wondering when it would become a national resource and then this pops up: GSA IDIQ RFP Hits the Street
Services for 3D Laser Scanning and Building Information Modeling (BIM)
Nationwide Laser Scanning Services
Here is the GSA view of BIM: Building Information Modeling (BIM)

And here is the core of the RFP:

1) Exterior & Interior 3D-Laser Scanning
2) Transform 3D laser scanning point cloud data to BIM models
3) Professional BIM Modeling Services based on 3D laser scanning data
    including, but not limited to:
   a. Architectural
   b. Structural
   c. MEP
   d. Civil
4) 3D Laser Scanning-based analysis & applications including,
     but not limited to:
   a. Visualization
   b. Clash Detection
   c. Constructability analysis
5) Integration of multiple 3D Laser Scanning Data, BIMs & analyses
6) Integration between multiple 3D Laser Scanning Data and BIM softwares
7) 3D Laser Scanning Implementation Support, including:
   a. Development of 3D Laser Scanning assessment and project
       implementation plans
   b. Selection of VDC software
   c. Support to project design and construction firms
   d. Review of BIM models & analysis completed by other
       service providers
8  Development of Best Practice Guidance, Case Studies, Procedures
     and Training Manuals
9) Development of long- & short-term 3D Laser Scanning implementation
      strategies,
10) Development of benchmarking and measurement standards and programs
11) 3D Laser Scanning and Modeling Training, including:
   a. Software specific training
   b. Best Practices

This is aimed at creating a national building inventory rather than a terrain model, but still very interesting. The interior/exterior ‘as built’ aspect has some novelty to it. Again virtual and real worlds appear to be on a collision trajectory and may soon overlap in many interesting ways. How to make use of a vast BIM modeling infrastructure is an interesting question. The evolution of GE and VE moves inexorably toward a browser virtual mirror of the real world. Obviously it is useful to first responders such as fire and swat teams, but there may be some additional ramifications. Once this data is available will it be public? If so how could it be adapted to internet use?

Until recently 3D modeling in browsers was fairly limited. Google and Microsoft have both recently provided useful api/sdk libraries embeddable inside a browser. The terrain model in GE and VE is vast but still relatively coarse and buildings were merely box facades in most cases. However, Johannes Kebek’s blog points to a recent VE update which re-enabled building model interiors, allowing cameras to float through interior space. Following a link from Kebek’s blog to Virtual Earth API Release Information, April 2009 reveals these little nuggets:

  • Digital Elevation Model data – The ability to supply custom DEM data, and have it automatically stitched into existing data.
  • Terrain Scaling – This works now.
  • Building Culling Value – Allows control of how many buildings are displayed, based on distance and size of the buildings.
  • Turn on/off street level navigation – Can turn off some of the special effects that occur when right next to the ground.

Both Google and Microsoft are furnishing modeling tools tied to their versions of virtual online worlds. Virtual Earth 3dvia technology preview appears to be Microsoft’s answer to Google Sketchup.
The race is on and shortly both will be views into a virtual world evolving along the lines of the gaming markets. But, outside of the big two, GE and VE, is there much hope of an Open Virtual World, OVM? Supposing this BIM data is available to the public is there likely to be an Open Street Map equivalent?

Fortunately GE and VE equivalent tools are available and evolving in the same direction. X3D is an interesting open standard scene graph schema awaiting better implementation. WPF is a Microsoft standard with some scene building capability which appears to be on track into Silverlight … eventually. NASA World Wind Java API lets developers build applet views and more complex Java Web Start applications which allow 3D visualizing. Lots of demos here: World Wind Demos
Blender.org may be overkill for BIM, but is an amazing modeling tool and all open source.

LiDAR Terrain

Certainly BIM models will find their way into browsers, but there needs to be a corresponding evolution of terrain modeling. BIM modeling is in the sub foot resolution while terrain is still in the multimeter resolution. I am hopeful that there will also be a national resource of LiDAR terrain data in the sub meter resolution, but this announcement has no indication of that possibility.

I’m grappling with how to make use of the higher resolution in a web mapping app. GE doesn’t work well since you are only draping over their coarse terrain. Silverlight has no 3D yet. WPF could work for small areas like architectural site renderings, but it requires an xbap download similar to Java Web Start. X3D is interesting, but has no real browser presence. My ideal would be something like an X3D GeoElevationGrid LOD pyramid, which will not be available to browsers for some years yet. The new VE sdk release with its ability to add custom DEM looks very helpful. Of course 2D contours from LiDAR are a practical use in a web map environment until 3D makes more progress in the browser.

Isenburg’s work on streaming meshes is a fascinating approach to reasonable time triangulation and contouring of LiDAR points. Here is an interesting kml of Mt St Helen’s contours built from LAS using Isenburg’s streaming methods outlined in this paper: Streaming Computation of Delaunay Triangulations.

Mt St Helen's Contours
Fig 2 – Mt St Helen’s Contours generated from streaming mesh algorithm

A national scale resource coming out of the “Stimulus” package for the US will obviously be absorbed into GE and VE terrain pyramids. But neither offers download access to the backing data, which is what the engineering world currently needs. What would be nice is a national coverage at some submeter resolution with the online tools to build selective areas with choice of projection, contour interval, tin, or raw point data. Besides, Architectural Engineering documents carry a legal burden that probably excludes them from online connectivity.

A bit of calculation (storage only) for a national DEM resource :
1m pixel resolution
AWS S3 $0.150 per GB – first 50 TB / month of storage used = $150 tb/month
US = 9,161,923,000,000 sq m @ 4bytes per pixel => 36.648 terabytes = $5497.2/month

0.5m pixel resolution
$0.140 per GB next 50 TB / month of storage used = $140 tb/month
US = 146.592 terabytes = $20,522/month

Not small figures for an Open Source community resource. Perhaps OVM wouldn’t have to actually host the data at community expense, if the government is kind enough to provide WCS exposure, OVM would only need to be another node in the process chain.

Further into the future, online virtual models of the real world appear helpful for robotic navigation. Having a mirrored model available short circuits the need to build a real time model in each autonomous robot. At some point a simple wireless connection provides all the information required for robot navigation in the real world with consequent simplification. This of course adds an unnerving twist to some of the old philosophical questions regarding perception. Does robotic perception based on a virtual model but disconnected from “the real” have something to say to the human experience?

X3D and Virtual Scene Graphs

x3d earth
Fig 1 – XJ3D Viewer showing an x3d earth model.

Virtual Worlds and Scene Graphs

Well I confess, I’m no gamer! I’ve never handled a Wii or blown away monsters on a PS3. It doesn’t hold my interest that much. Guitar Hero makes me feel slightly nauseous. (When will they introduce Symphony Conductor for the Wii?) I suppose I’m an anachronism of the old paperback book era. My kids groan, but I can still get excited about the E.E. Smith Lensman series. However, I’m a bit chagrined to admit that I was oblivious to X3D until this past week. It’s just that gaming and GIS have not overlapped in my world yet.

X3D has been around quite awhile. Some of the specs date back as far as 2003 and then there is the VRML97 ancestory so this is not new stuff. It reminds me a bit of the early SVG days, circa 2000. The novelty of event driven graphics in the browser was overpowering then, but it wasn’t until years later that Flash and Silverlight actually made xml graphics generally available to web app interfaces, almost ten years later. Now of course, SVG even has native support in most of the non microsoft browsers. I can imagine in the year 2018 that 3D graphics XML will be ubiquitous, but it will probably be some derivative of ChromeFlash, XNA, or WPF, while X3D may still languish in the outer reaches of civilization.

X3D has a similar feel in that it’s a powerful anticipating specification with beautiful XML scene graph capability, but only a few implementations far from the mainstream. I won’t say I’m a sucker for XML, but I really, really like to look at a file and understand what it says. Nothing beats a good old text editor for playing with an XML graphics file. Sorry, my text bias is showing again.

The Java3D API is a parallel enabling technology. The scene graph approach implemented in Java3D’s api enabled implementations of free X3D viewers. There are several viewers, but the one that actually worked on the geospatial component of X3D was Xj3D.

What is X3D?

Directly from the x3d site:

“X3D is a scalable and open software standard for defining and communicating real-time, interactive 3D content for visual effects and behavioral modeling. It can be used across hardware devices and in a broad range of applications including CAD, visual simulation, medical visualization, GIS, entertainment, educational, and multimedia presentations.”

“X3D provides both the XML-encoding and the Scene Authoring Interface (SAI) to enable both web and non-web applications to incorporate real-time 3D data, presentations and controls into non-3D content.”

and here is an example of what it looks like:
  
   <?xml version=”1.0″ encoding=”UTF-8″?>
   <!DOCTYPE X3D PUBLIC “ISO//Web3D//DTD X3D 3.2//EN”
   “http://www.web3d.org/specifications/x3d-3.2.dtd”>
   <X3D
   profile=”Immersive”
   version=”3.2″
   xmlns:xsd=”http://www.w3.org/2001/XMLSchema-instance”
   xsd:noNamespaceSchemaLocation=”http://www.web3d.org/specifications/x3d-3.2.xsd”>
      <head>
         <component name=”Geospatial” level=”1″/>
      </head>
      <Scene>
         <Group>
            <PointLight location=”10000000000 0 0″ radius=”100000000000000″/>
            <PointLight location=”-10000000000 0 0″ radius=”100000000000000″/>
            <PointLight location=”0 10000000000 0″ radius=”100000000000000″/>
            <PointLight location=”0 -10000000000 0″ radius=”100000000000000″/>
            <PointLight location=”0 0 10000000000″ radius=”100000000000000″/>
            <PointLight location=”0 0 -10000000000″ radius=”100000000000000″/>
  
            <GeoViewpoint
             position=”39.755092 -104.988123 60000″
             orientation=”1.0 0.0 0.0 -1.57″
             description=”Denver”
             headlight=”true”
             fieldOfView=”1.57″
             geoSystem=’”GDC”‘>
             <GeoOrigin DEF=”ORIGIN”
             geoCoords=”0.0 0.0 -6378137″
             geoSystem=”GDC” />
            </GeoViewpoint>
            <Inline url=”tiles/0/tile_0.x3d”/>
         </Group>
      </Scene>
   </X3D>

After the normal XML declaration follows the root element:
   <X3D
   profile=”Immersive”
   version=”3.2″
   xmlns:xsd=”http://www.w3.org/2001/XMLSchema-instance”
   xsd:noNamespaceSchemaLocation=”http://www.web3d.org/specifications/x3d-3.2.xsd”>

In this case using version 3.2 of the X3D spec and a profile, “Immersive”. Profiles are used to break the X3D spec into smaller implementation areas. Consequently, with less effort, an implementor can meet a profile of the X3D spec suitable to the need instead of the entire “Full” specification.

X3D Profiles
Fig 2 – X3D Profiles

Curiously, the X3D Full profile lists the geospatial component, but the Immersive profile seemed to implement the geospatial elements of interest in my particular case. Perhaps geospatial was later moved to the Full profile. Here is a list of the xml node elments found in the geospatial component:

  • GeoCoordinate
  • GeoElevationGrid
  • GeoLocation
  • GeoLOD
  • GeoMetadata
  • GeoOrigin
  • GeoPositionInterpolator
  • GeoProximitySensor
  • GeoTouchSensor
  • GeoTransform
  • GeoViewpoint

You can see a ‘GeoViewpoint’ element and a ‘GeoOrigin’ element in the small x3d listing above. The Immersive/Full profile is meant to imply a virtual world immersion experience. This means that a scene graph populates a world with context including lighting and a viewpoint that navigates through the virtual world space. Now this is of interest to a GIS developer in the sense that the virtual world space can mimic a real world space.

X3D Earth provides some nice examples of using X3D with earth data. Not all of the examples are in working order using the latest build of XJ3D Browser. These are older samples and may not reflect the current configuration of the X3D specification.

Another example of a virtual world space mimicing the actual world is Google’s Earth API. You can use a flying viewpoint to navigate through a model of the Rocky Mountains and get something like the experience of a bird or fighter jet. The mimicry is still coarse but the effect is recognizable. This is the horizon end point of mapping. The map abstraction eventually disappears into a one to one mathematical mapping with some subtle abstraction available as overlays and heads up displays.

This is where the online mapping world eventually merges with the gaming world. Only for mapping, the imaginary part of the world is de-emphasized in favor of an immersive real world experience as an aid to navigating in external physical space. Thinking about these kinds of worlds and their overlaps has proven fascinating, as shown by the popularity of the Matrix movie series. Are we the Ghost in the Machine or the other way around? But that is best left to philosophy types.

Let’s move on to a couple of other useful X3D scene elements. GeoLOD geospatial Level Of Detail element is an interesting element for use in geospatial worlds. By recursively referencing deeper levels of x3d GeoTerrain tiles you have an affect similar to DeepEarth tile pyramids, but with the addition of 3D Terrain:

<?xml version=”1.0″ encoding=”UTF-8″?>
<!DOCTYPE X3D PUBLIC “ISO//Web3D//DTD X3D 3.2//EN”
“http://www.web3d.org/specifications/x3d-3.2.dtd”>
<X3D
profile=”Immersive”
version=”3.2″
xmlns:xsd=”http://www.w3.org/2001/XMLSchema-instance”
xsd:noNamespaceSchemaLocation=”http://www.web3d.org/specifications/x3d-3.2.xsd”>
   <head>
      <component name=”Geospatial” level=”1″/>
      </head>
      <Scene>
         <GeoLOD
            center=”-61.875 -168.75 0.0 ”
            range=’2507443.095376239′
            geoSystem=”GDC”
            child1Url=’../6/tile_0000100.x3d’
            child2Url=’../6/tile_0000101.x3d’
            child3Url=’../6/tile_0000102.x3d’
            child4Url=’../6/tile_0000103.x3d’
         >
         <GeoOrigin DEF=”ORIGIN”
            geoCoords=”0.0 0.0 -6378137.0 ”
            geoSystem=”GDC” />
         <Group containerField=’rootNode’>
            <Shape>
               <Appearance>
                  <Material diffuseColor=”0.3 0.4 0.3 “/>
                  <ImageTexture
                     url=”../../images/5/img_000010.png”
                     repeatS=”false”
                     repeatT=”false”/>
               </Appearance>
                     <GeoElevationGrid
                        ccw=’true’
                        colorPerVertex=’false’
                        creaseAngle=’0′
                        geoGridOrigin =”-67.5 -180.0 0.0 ”
                        yScale =’1.0′
                        geoSystem=’”GDC”‘
                        normalPerVertex=’false’
                        solid=’true’
                        xDimension=’37′
·                     xSpacing=’0.625′
                        zDimension=’37′
                        zSpacing=’0.3125′
                        height=”
                        -1618 -3574 -3728 -3707 -3814 -3663 -3455 -3089
                         -3258 -3457 -3522 -3661 -3519 -3757 -3412 -3379
                         -3674 -3610 -3503 -3679 -3521 -3537 -3397 -3790
                         -3795 -3891 -3883 -3864
                                             .
                                             .
                         -5086 -5138 -5089 -5229 -4965 -5021 -5064 -5099
                         -5062 -4952 -4955 -4810 -4635 -4686 -4599 -4408
                         -4470 -5407 -4482 -4600 -4486 -3942 -4202 -4100
                         -4306 -4293 -4010 -4091
                         “>
                      <GeoOrigin USE=”ORIGIN” />
                  </GeoElevationGrid>
               </Shape>
            </Group>
         </GeoLOD>
   </Scene>
</X3D>

In the x3d snippet above you can see the GeoLOD that references down to level 6 of the terrain tile hierarchy. This is a recursive reference as each tile lower in the pyramid references the four tiles of the next level that make up the current tile, basically a quad tree approach found in most tile setups. The element attributes include a center, range, geosystem, and the ChildURL references. Inside the GeoLOD is a <Group> <Shape> element containing the current tile’s Appearance Material with a reference to the draping tile image and a GeoElevationGrid element with an array of elevations relative to the GeoOrigin.

For this to work there are two pyramids, one containing the draping images and one with the x3d GeoTerrainGrids. The GeoTerrainGrid is the same as x3d ElevationGrid except it introduces the concept of a geospatial reference system. This is all very clever and useful, but it requires the Xj3D Browser viewer to play with it. In order to author this kind of world scene you also need tools to tile both the underlying terrain models as well as the overlaid imagery.

x3d earth
Fig 3 – x3d earth model tile pyramid

Over in Microsoft I took a quick look at XNA Game Studio, which is the current development tool for XBox games. The XNA serializer supports a type of XML <XnaContent>, but there doesn’t seem to be much information about the XML XnaContent schema. I wasn’t able to find an X3D importer for XNA. Too bad, since an XNA importer would be another way to experiment with Terrain grids.

Summary:

As far as the future, it is easy to see that the WPF <MeshGeometry3D> can be used to perform the duties of a an ElevationGrid element. This means, at least in WPF, the basic building blocks are available to display and manipulate a scene in 3D. What is lacking is the concept of LOD, Level of Detail. After first seeing the DeepZoom MultiScaleImage element in action, I fell in love with the very natural navigation through its 2D space.

It is not much of a stretch to see this being extended to a “MultiScaleMeshGeometry3D” element. Again WPF appears to be a look at the future of Silverlight, so it is not impossible to imagine this type of capability in browsers by 2018. In the non-Microsoft world X3D will play a role once implementations make their way into Chrome, FireFox, and Opera. In some form or other, XML scene graphs will be in the future of GIS, and yes, we will all be pro gamers!

BMNG SilverLight 2.0 Beta Deep Zoom

Well somebody had to do it! Im willing to give it a try.

What is it?

One of the many announcements at the MIX08 conference is the availability of Deep Zoom technology for Silverlight 2.0 Beta 1. This results from an R&D program in Microsoft called Sea Dragon. Sea Dragon was evidently a Microsoft acquisition awhile back. Reminiscent of Keyhole(now google earth), Sea Dragon is a tool for smooth viewing of very large image resources. The novelty is to have it useable inside a SilverLight browser view. http://labs.live.com/Seadragon.aspx

For those so inclined to load beta stuff there is a demo available here with a nice code behind zoom pan capability: http://visitmix.com/blogs/Joshua/Hard-Rock-Cafe/

The idea is that behind the viewport is a large image in PNG, JPEG, TIFF, BMP format which is fed to the screen resolution using a MultiScaleImage element in a SilverLight 2.0 page. Actually just jpeg as it turns out.

Well of course this is extremely interesting to geospatial people since all of the WMS, WCS, WFS, KML, tile caching, image pyramids etc are aimed at exactly this functionality.

How is the user experience?

Holy Mackerel, finally a reason to use Vista! But wait this is SilverLight it should work in FireFox and Safari too. By the way,if any Sea Dragon big Tuna comes trolling by this post I already have my map hacker wishlist for 2.0 at the end of this entry.

OK, let’s try it.

First I grabbed a full resolution BMNG image off of BitTorrent: http://www.geotorrent.org/torrents/61.torrent

Chris Holmes has detailed instructions on this part here: http://docs.codehaus.org/display/GEOSDOC/Load+NASA+Blue+Marble+Data

The eventual result will be twelve large(267Mb) ecw files, “world-topo-bathy-200408-3x86400x43200.ecw,” one for each month of 2004. Ecw is a wavelet compression format for imagery http://en.wikipedia.org/wiki/ECW_(file_format), but to use the image we need it in a different format, Gdal to the rescue. The easiest approach is to take advantage of the FWtools bin directory to run a command line translation like this:

“C:\Program Files\FWTools2.1.0\bin\gdal_translate” -of GTiff world-topo-bathy-200401-3x86400x43200.ecw BMNG.tiff

After a few minutes the result is a tiff image of 86400×43200 of about 11Gb. Now it is time to use the Deep Zoom Composer (actually a decomposer) to process this into a MultiScaleImage info.bin
http://blogs.msdn.com/expression/archive/2008/03/05/download-the-preview-of-the-deep-zoom-composer.aspx

When I attempted an import of this 11Gb tiff into Deep Zoom Composer, the Mermaid.exe choked after a few minutes. I guess we aren’t ready for geospatial scale exactly yet. Note to self: do this with -o Tiff, since mermaids may not like GTiff.

So I went back to a smaller downsample to start my experiment. This time I chose a 3600×1800 png at 4.154Mb. This was rather instant success. Now there is a BMNG test pyramid on my hard drive. The pyramid is 13 levels and each subdirectory contains the necessary images in jpg. Deep Zoom Composer rather handily chunks the tiles in each level, even calculating non square tiling.


Fig 1 Example of the a pyramid level resulting from Deep Zoom Image Composer

After playing around a bit the export spits out what we need for a Silverlight MultiScaleImage. Remember this is an element type introduced with SilverLight2.0 Beta so you can’t really see this unless you want to do a beta Silverlight 2.0 install.

Here are some links on other neat things in R&D over at Microsoft labs.live: http://labs.live.com/photosynth/whatis/

SilverLight works a lot better with an IIS server, but I am using an Apache server, so I created a javascript Silverlight project. Using the default project, I modified the Scene.xaml and associated scene.js to make use of the new MultiScaleImage element:

<MultiScaleImage      
x:Name=msi“      
ViewportWidth=1.0“      
ViewportOrigin=0,0“      
Source=/BMNGZoom/BMNG/info.bin>

This worked pretty well to get the image object in the browser with a sort of spring loaded entre. Perhaps the springy annoyance can be turned off by setting UseSpringsProperty=”off.” However, adding zoom and pan are bit more problematic. I am brand new to Silverlight but oddly there seem to be very few events available:
MouseMove, MouseEnter, MouseLeave, MouseLeftButtonDown, MouseLeftButtonUp

If you want MouseRight, Keyboard, MouseWheel etc, you need to have some code behind. Since I didn’t really have time to figure out all of the code behind tricks for getting this to serve from Apache, I took a primitive approach. By attaching an event to MouseLeftButtonUp I can simulate a click event. Then connecting this click event to the MultiScaleImage ViewportWidth *= 0.9; I could make a one way zoom in without too much effort. Not very useful, but good enough to get a feel for the interaction, which by the way is very impressive. The zooming is familiar to anyone used to VE or GE type of continuous zoom. Pretty nifty for a browser interface.

There is even an ‘old view’ to ‘new view’ animation effect, which cleverly distracts the user while the new tiles are streaming in over the old tiles. Local tile cache makes revisiting very smooth. I will have to try this on an older tier 0 GPU system so I can watch the tiles move slowly into place.

http://www.web-maps.com/BMNGZoom/ >


Fig 2- primitive test of a Deep Zoom BMNG

Now that I had this working on a relatively small 4.1 Mb image, my next step was to step up the size and see what the effect would be. I already knew 11Gb Gtiff was not going to work. Dividing the full Blue Marble into 8 tiles and using PNG output seemed like a possibility. This gives 8 files at 256Mb each.

However, I noticed that the pyramid files are jpeg so why not start with 8 jpeg files instead:
“C:\Program Files\FWTools2.1.0\bin\gdal_translate” -of JPEG
-projwin -180 90 -90 0 world-topo-bathy-200401-3x86400x43200.ecw BMNG1.jpg
Input file size is 86400, 43200
Computed -srcwin 0 0 21600 21600 from projected window.
0…10.

After a few minutes I could open the Deep Zoom Image Composer again and do an import on the full eight tile collection. The composer did not show anything in the view window with these larger jpg images so I was working blind on composition. I did the export anyway out of curiosity.
I’ll post the results next week since it will take a good bit of uploading time.
The result of this bit of experiment was quite surprising. The pyramid building goes fairly smoothly in the Deep Zoom Composer and is painless compared to manually building pyramids in Java. Actually Geotools ImagePyramid has some advantages like iteration over multiple images and command line capability. But the resulting tile pyramid doesn’t have the client side streaming.The MultiScaleImage element hides the complexity of an ajax slippy map interface in a single element. On the down side adding functionality seems to be aimed at IIS ASP type servers. I imagine with a bit of time I can work out the details of a pan and MouseWheel zoom. SilverLight includes RenderTransform matrix capability, it just requires code behind to make it useful with mouse and keyboard functions.

The question is “how does this work?” Of course the answer is “I dont know,” but that doesn’t stop some speculation. The pyramid is obvious. The fact that it works on both a linux or a windows box eliminates a stub socket on the server side. It appears to be an object downloaded to the client which orchestrates things in an ajax mode. Of course clr won’t work with Firefox on Linux or Safari so there must be a plugin object which can be duplicated cross platform.

Wishlist for Deep Zoom 2.0 from a map hacker

1. Can we scale this puppy to use extremely large image sets? I imagine DCOG is going to want to see their 6″ aerials for the Denver metro area in Deep Zoom. Why should they have to come up with a pyramid tile set of MultiScaleImage elements?

2. How about MultiScaleImage elements for WPF xaml/xbap? I would like to use it with a more comprehensive set of code behind tools as an xbap.

3. Once it’s in WPF how about using some OO magic and add MultiScaleImageBrush for draping on a 3D mesh?

Lets extrapolate a couple more steps

4. Why stop at images? How about extending to MultiScale3DMesh. Then my spare time project for an AJAX LiDAR viewer won’t require much work.

5. Don’t stop there, lets have MultiScaleImageBrush on MultiScale3DMesh.

Now sharpen your patent pen

6. Why not MultiScaleVideo? Sooner or later all of the bifocaled baby boomers will be downloading movies to their iPhone, oops ZunePhone. How else are we going to see anything on those miniscule screens. Besides talk about “immersive,” movies could really be interactive. Imax resolution on a phone, why not!

Wide Area HVAC controller using WPF and ZigBee Sensor grid

One project I’ve been working on recently revolves around an online controller for a wide area HVAC system. HVAC systems can sometimes be optimized for higher efficiency by monitoring performance in conjunction with environment parameters. Local rules can be established for individual systems based on various temperatures, humidity, and duct configurations. Briefly, a set of HVAC functions consisting of on/off relay switches and thermistors, can be observed from an online monitoring interface. Conversely state changes can be initiated online by issuing a command to a queue. These sensors and relays might be scattered over a relatively large geographic area and in multiple locations inside a commercial building.

It is interesting to connect a macro geospatial world with a micro world, drilling down through a local facility to a single thermistor chip. In the end its all spatial.

Using a simple map view allows drill down from a wide area to a building, a device inside a building, a switch bank, and individual relay or analog channel for monitoring or controlling. The geospatial aspect of this project is somewhat limited, however, the zoom and pan tools used in the map location also happen to work well in the facilities and graphing views.

The interface can be divided into three parts:
1) The onsite system – local base system and Zigbee devices
2) The online server system – standard Apache Tomcat
3) The online client interface – WPF xbap, although svg would also work with a bit more work

Onsite System

The electronically impaired, like myself, may find the details of controller PIC chip sets, relays, and thermistor spec sheets baffling, but really they look more hacky than they are:

Fig 0 – Left: Zigbee usb antenna ; Center: thermistor chip MCP9701A; Right: ProXR Zigbee relay controller

The onsite system is made up of sensors and controller boards. The controller boards include a Zigbee antenna along with a single bank of 8 relays and an addition set of 8 analog inputs. The sensors are wired to the controller board in this development mode. However, Zigbee enabled temperature sensors are also a possibility, just more expensive. See SunSpot for example: http://www.sunspotworld.com/ (Open Source hardware? )

ZigBee is a wifi type communications protocol based on IEEE 802.15.4. It allows meshes of devices to talk to each other via RF as long as they are within about 100-300 ft of another node on the mesh. Extender repeaters are also available. ZigBee enabled devices can be scattered around a facility and communicate back to a base system by relaying messages node to node through an ad hoc mesh network.

The onsite system has a local pc acting as the base server. The local onsite server communicates with an external connection via a internet router and monitors the Zigbee network. ASUS EeePCs look like a good candidate for this type of application. Messages originating from outside are communicated down to the individual relay through the ZigBee mesh, while state changes and analog readings originating from a controller relay or sensor channel are communicated up the ZigBee network and then passed to the outside from the local onsite server.

The local server must have a small program polling the ZigBee devices and handling communications to the outside world via an internet connection. The PC is equipped with a usb ZigBee antenna to communicate with the other ZigBee devices in the network. This polling software was written in Java even though that may not the best language for serial USB com control in Windows. The target system will be Linux based. The ZigBee devices we selected came with the USB driver that treats a USB port like a simple COM port.

Since this was a Java project the next step was finding a comm api. The sun JavaComm has discontinued support of Windows, although it is available for Linux. Our final onsite system will likely be Linux for cost reasons, so this is only a problem with the R&D system which is Windows based. I ended using a RXTX library, RXTXcomm.jar, at http://www.jcontrol.org/download/rxtx_en.html

Commands for our ProXR controller device are a series of numeric codes, for example<254;140;3;1>This series of commands puts the controller in command mode 254, a set bank status command 140, a byte indicating relays 0 and 1 on 3, and bank address 1. The result is relays 0 and 1 are switched to the on position. The commands are issued similarly for reading relay state and analog channels. <254;166;1> for example reads all 8 analog I/O channels as a set of 8 bytes.

Going in prototype mode we picked up a batch of three wire MCP9701A thermistor chips for a few dollars. The trick is to pick the right resistance to get voltage readings in to the mid range of the 8bit or 10bit analog channel read. Using 8 bit output lets us poll for temperature with around .5 degree F resolution.

The polling program issues commands and reads results on separate threads. If state is changed locally it is communicated back to the online server on the next polling message, while commands from the online command queue are written to the local controller boards with the return. In the meantime every polling interval sends an analog channel record back to the server.

Online Server

The online server is an Apache Tomcat service with a set of servlets to process communications from the onsite servers. Polled analog readings are stored in a PostgreSQL database with building:device: bank:channel addresses as well as a timestamp. The command queue is another PostgreSQL table which is checked at each poll interval for commands addressed to the building address which initiated the poll. Any pending commands are returned to the polling onsite server where they will be sent out to the proper device:bank:relay over the ZigBee network.

Two other tables simply provide locations of buildings as longitude, latitude in the wide area HVAC control system. Locations of devices insidebuildings are stored in a building table as floor and x,y coordinates. These are available for the client interface.

Client Interface

The client interface was developed using WPF xbap to take advantage of xaml controls and a WMS mapping interface. Initially the client presents a tabbed menu with a map view. The map view indicates the wide area HVAC extents with a background WMS image for reference. Zooming in to the building location of interest allows the user to select a building to show a floor plan with device locations indicated.

Fig 1 HVAC wide area map view

Once a building is selected the building floor plans are displayed. Selecting an individual device determines the building:device address.

Fig 2 Building:device address selection from facilities floor plan map

Finally individual relays can be selected from the device bank by pushing on/off buttons. Once the desired switch configuration is set in the panel, it can be sent to the command queue as a building:device:bank address command. Current onsite state is also double checked by the next polling return from the onsite server.

Fig 3 Relay switch panel for selected building:device:bank address.

The analog IO channels are updated to the server table at the set polling interval. A selection of the analog tab displays a set of graph areas for each of the 8 channels. The on/off panel initiates a server request for the latest set of 60 polled readings which are displayed graphically. It won’t be much effort to extend this analog graph to a bidirectional interface with user selectable ranges set by dragging floor and ceiling lines that trigger messages or events when a line is crossed.

Fig 4 Analog IO channel graphs

This prototype incorporates several technologies using a Java based Tomcat service online and a Java RXTXcomm Api for the local Zigbee polling. The client interface is also served out of Apache Tomcat as WPF xaml to take advantage of easier gui control building. In addition OGC WMS is used for the map views. The facilities plan views will be DXF translations to WPF xaml. Simple graphic selection events are used to build addresses to individual relays and channels. The server provides historical command queues and channel readings by storing time stamped records. PostgreSQL also has the advantage of handling record locking on the command queue when multiple clients are accessing the system.

This system is in the prototype stage but illustrates the direction of control systems. A single operator can maintain and monitor systems from any locations accessible to the internet, which is nearly anywhere these days. XML rendering graphics grammars for browsers like svg and xaml enable sophisticated interfacesthat are relatively simple to build.

There are several OGC specifications oriented toward sensor grids, http://www.opengeospatial.org/projects/groups/sensorweb. The state of art is still in flux but by virtue of the need for spatial management of sensor grids, there will be a geospatial component in an “ubiquitous sensor” world.

LiDAR processing with Hadoop on EC2

I was inspired yesterday by a FRUGOS talk from a Lidar production manager showing how to use some OpenSource tools to make processing LiDAR easier. LiDAR stands for LIght Detection And Ranging. It’s a popular technology these days for developing detailed terrain models and ground classification. The USGS has some information here: USGS CLick and the state of PA has an ambitious plan to fly the entire state every three years at a 1.4 ft resolution. PAMAP LiDAR

My first stop was to download some .las data from the PAMAP site. This PA LiDAR has been tiled into 10000x10000ft sections which are roughly 75Mb each. First I wrote a .las translator so I could look at the data. The LAS 1.1 specification is about to be replaced by LAS 2.0 but in the meantime most available las data is still using the older spec. The spec contains a header with a space for variable length attributes and then the point data. Each point record contains 20bytes (at least in the PAMAP .las)  of information including x,y,z, as long values and a classification and reflectance value. The xyz data is scaled and offset by values in the header to obtain double values. The LiDAR sensor mounted on an airplane flies tracks across the are of interest while the LiDAR pulses at a set frequency. The location of the reading can then be determined by post processing against the GPS location of the instrument. The pulse scans result in tracks that angle across the flight path.

Since the PAMAP data is available in PA83-SF coordinates, despite the metric unit shown in the meta data, I also added a conversion to UTM83-17. This was a convenience for displaying over Terraserver backgrounds using the available EPSG:26917. Given the hi res aerial data available from PAMAP this step may be unnecessary in a later iteration.

Actually PAMAP has done this step already. Tiff grids are available for download. However, I’m interested in the gridding algorithm and possible implementation in a hadoop cluster so PAMAP is a convenient test set with a solution already available for comparison. It is also nice to think about going directly from raw .las data to my end goal of a set of image and mesh pyramids for the 3D viewer I am exploring for gmti data.


Fig 1 – LiDAR scan tracks in xaml over USGS DOQ

Although each section is only about 3.5 square miles the scans generate more than a million points per square mile. The pulse rate achieves roughly 1.4ft spacing for the PAMAP data collection. My translator produces a xaml file of approximately 85Mb which is much too large to show in a browser. I just pulled some scans from the beginning and the end to show location and get an idea of scale.

The interesting aspect of LiDAR is the large amount of data collected which is accessible to the public. In order to really make use of this in a web viewer I will need to subtile and build sets of image pyramids. The scans are not orthogonal so the next task will be to grid the raw data into an orthoganal cell set. Actually there will be three grids, one for the elevation, one for the classification, and one for the reflectance value. With gridded data sets, I will be able to build xaml 3D meshes overlaid with the reflectance or classification. The reflectance and classification will be translated to some type of image file, probably png.

Replacing the PA83-SF with a metric UTM base gives close to 3048mx3048m per PAMAP section.If the base cell starts at 1 meter, and using a 50cellx50cell patch, the pyramid will look like this:
dim cells/patch size  approx faces/section
1m 50x50  50mx50m  9,290,304
2m 50x50  100mx100m 2,322,576
4m 50x50  200mx200m   580,644
8m 50×50   400mx400m   145,161
16m 50×50   800mx800m 36,290
32m 50x50  1600mx1600m 9072
64m 50x50  3200mx3200m 2268
128m 50x50  6400mx6400m 567
256m 50x50  12800mx12800m 141

There will end up being two image pyramids and a 3D zaml mesh pyramid with the viewer able to move from one level to the next based on zoom scale. PAMAP also has high resolution aerial imagery which could be added as an additional overlay.

Again this is a large amount of data, so pursuing the AWS cluster thoughts from my previous posting it will be interesting to build a hadoop hdfs cluster. I did not realize that there is actually a hadoop public ami available with ec2 specific scripts that can initialize and terminate sets of EC2 instances. The hadoop cluster tools are roughly similar to the Google GFS with a chunker master and as many slaves as desired. Hadoop is an Apache project started by the Lucene team inspired by the Google GFS. The approach is useful in processing data sets larger than available disk space, or 160Gb in a small EC2 instance. The default instance limit for EC2 users is currently at 20. With permission this can be increased but going above 100 seems to be problematic at this stage. Using a hadoop cluster of 20 EC2 instances still provides a lot of horsepower for processing data sets like LiDAR. The cost works out to 20x$0.10 per hour = $2.00/hr which is quite reasonable for finite workflows. hadoop EC2 ami

This seems to be a feasible approach to doing gridding and pyramid building on large data resources like PAMAP. As PAMAP paves the way, other state and federal programs will likely follow with comprehensive sub meter elevation and classification available for large parts of the USA. Working out the browser viewing requires a large static set of pre-tiled images and xaml meshes. The interesting part of this adventure is building and utilizing a supercomputer on demand. Hadoop makes it easier but this is still speculative at this point.


Fig 1 – LiDAR scan tracks in xaml over USGS DRG

I still have not abandoned the pipline approach to WPS either. 52North has an early implementation of a WPS available written in Java. Since it can be installed as a war, it should be simple to startup a pool of EC2 instances using an ami with this WPS war installed. The challenge, in this case, is to make a front end in WPF that would allow the user to wire up the instance pool in different configurations. Each WPS would be configured with an atomistic service which is then fed by a previous service on up the chain to the data source. Conceptually this would be similar to the pipleline approach used in JAI which builds a process chain, but leaves it empty until a process request is initialized. At that point the data flows down through the process nodes and out at the other end. Conceptually this is quite simple, one AMI per WPS process and a master instance to push the data down the chain. The challenge will be a configuration interface to select a WPS atom at each node and wire the processes together.

I am open to suggestions on a reasonably useful WPS chain to use in experimentations.

AWS is an exciting technology and promises a lot of fun for computer junkies who have always had a hankering to play with supercomputing, but didn’t quite make the grade for a Google position.

Xaml on Amazon EC2 S3

Time to experiment with Amazon EC2 and S3. This site http://www.gis-xaml.com is using an Amazon EC2 instance with a complete open source GIS stack running on Ubuntu Gutsy.

  • Ubuntu Gutsy
  • Java 1.6.0
  • Apache2
  • Tomcat 6.016
  • PHP5
  • MySQL 5.0.45
  • PostgreSQL 8.2.6
  • PostGIS 1.2.1
  • GEOS 2.2.3-CAPI-1.1.1
  • Proj 4.5.0
  • GeoServer 1.6

Running an Apache2 service with a jk_mod connector to tomcat lets me run the examples of xaml xbap files with their associated java servlet utilities for pulling up GetCapabilities trees on various OWS services. This is an interesting example of combining open source and WPF. In the NasaNeo example Java is used to create the 3D terrain models from JPL srtm (Ctrl+click) and drape with BMNG all served as WPF xaml to take advantage of native client bindings. NasaNeo example

I originally attempted to start with a public ami based on fedora core 6. I found loading the stack difficult with hard to find RPMs and difficult installation issues. I finally ran into a wall with the PostgreSQL/PostGIS install. In order to load I needed a complete gcc make package to compile from sources. It did not seem worth the trouble. At that point I switched to an Ubuntu 7.10 Gutsy ami.

Ubuntu based on debian is somewhat different in its directory layout from the fedora base. However, Ubuntu apt-get was much better maintained than the fedora core yum installs. This may be due to using the older fedora 6 rather than a fedora 8 or 9, but there did not appear to be any useable public ami images available on the AWS EC2 for the newer fedoras. In contrast to fedora on Ubuntu installing a recent version of PostgreSQL/PostGIS was a simple matter:
apt-get install postgresql-8.2-postgis postgis

In this case I was using the basic small 32 bit instance ami with 1.7Gb memory and 160Gb storage at $0.10/hour. The performance was very comparable to some dedicated servers we are running, perhaps even a bit better since the Ubuntu service is setup using an Apache2 jk_mod to tomcat while the dedicated servers simply use tomcat.

There are some issues to watch for on the small ami instances. The storage is 160Gb but the partition allots just 10Gb to root and the balance to a /mnt point. This means the default installations of mysql and postgresql will have data directories on the smaller 10Gb partition. Amazon has done this to limit ec2-bundle-vol to a 10GB max. ec2-bundle-volume is used to store an image to S3 which is where the whole utility computing gets interesting.

Once an ami stack has been installed it is bundled and stored on S3, that ami is then registered with AWS. Now you have the ability to replicate the image on as many instances as desired. This allows very fast scaling or failover with minimal effort. The only caveat of course is in dynamic data. Unless provision is made to replicate mysql and postgresql data to multiple instances or S3, any changes can be lost with the loss of an instance. This does not appear to occur terribly often but then again the AWS is still Beta. Also important to note, the DNS domain pointed to an existing instance will also be lost with the loss of your instance. Bringing up a new instance requires a change to the DNS entry as well (several hours), since each instance creates its own unique amazon domain name. There appear to be some work arounds for this requiring more extensive knowledge of DNS servers.

In my case the data sources are fairly static. I ended up changing the datadir pointers to /mnt locations. Since these are not bundled in the volume creation, I handled them separately. Once the data required was loaded I ran a tar on the /mnt/directory and copied the .tar files each to its own S3 bucket. The files are quite large so this is not a nice way to treat backups of dynamic data resources.

Next week I have a chance to experiment with a more comprehensive solution from Elastra. Their beta version promises to solve these issues by wrapping Postgresql/postgis on the ec2 instance with a layer that uses S3 as the actual datadir. I am curious how this is done but assume for performance the indices remain local to an instance while the data resides on S3. I will be interested to see what performance is possible with this product.

Another interesting area to explore is Amazon’s recently introduced SimpleDB. This is not a standard sql database but a type of hierarchical object stack over on S3 that can be queried from EC2 instances. This is geared toward non typed text storage which is fairly common in website building. It will be interesting to adapt this to geospatial data to see what can be done. One idea is to store bounding box attributes in the SimpleDB and create some type of JTS tool for indexing on the ec2 instance. The local spatial index would handle the lookup which is then fed to the SimpleDB query tools for retrieving data. I imagine the biggest bottleneck in this scenario would be the cost of text conversion to double and its inverse.

Utility computing has an exciting future in the geospatial realm – thank you Amazon and Zen.

WPF Xaml Coexists with Java


Fig 1 Example of 2D WPF OWS Mapping Interface

One of the interesting aspects of XML is its decoupling affect. XML readers and writers can be language and OS agnostic. Adding AJAX to the mix creates some interesting technology interactions. Since Microsoft WPF Xaml is an XML rendering grammar, Xaml can be created using either a Microsoft approach, such as Windows, IIS, SqlServer, C#, ASP, or alternative platforms like LAMP, Linux, Apache, MySQL, PHP or in my case Linux, Apache, PostgreSQL, Java.

The Java language provides some flexibility to the server side. This becomes especially important as virtual systems and utility computing models become more popular. Amazon’s Elastic Compute Cloud, EC2 http://aws.amazon.com/, is an important platform for startups and small businesses. By keeping development in a Linux/Apache based platform a small business can easily take advantage of Amazon’s EC2 instances for low entry cost and then scale to meet traffic fluctuations. EC2/S3 lets the computing platform instances become cash flow dependent rather than a fixed capital investment.

Based on Linux, EC2 is not available for Microsoft technology and servers. Although Microsoft provides Silverlight streaming service http://silverlight.live.com, it is not marketing it as broadly and extensively as Amazon’s EC2/S3 service. Setting up EC2 instances with Mono based C# might be feasible, but adds complexity and is not likely to keep up with the newest tools and technology from Microsoft. Depending on Amazons success with utility computing other large players such as Microsoft, Google, IBM, and governments could enter the utility computing infrastructure field. In the meantime AWS is a significant reason to keep web server options open to Linux/Apache.

The rise of XML makes bridging all the various platforms much easier and robust. WPF Xaml/Xbap applications can be built using Visual Studio and served from any service platform. For instance a WPF XBAP service can be compiled on a Microsoft VS development system, but then served from a Linux server using Apache Tomcat and Java. This means that I can develop the client side using WPF C# taking advantage of all the rich client graphic and 3D capability. I can then create an Xbap for publishing from an Apache web server running on a Linux platform, which in fact can be a virtual EC2 instance. Server side intelligence can take advantage of Java’s flexibility and power.

On the client side, WPF’s power as a rendering engine is extending beyond IE. As .NET rolls up to version 3.5 additional browser support has been added for FireFox and Safari, subsuming the main browser platforms in the world at large. WPF adds another benefit, instead of depending so heavily on CPUs at the server for typical web applications based on ASP, JSP, PHP, Dojo, Java Server Faces etc, the client CPU is pulled into action distributing the workload out across the internet. In spite of the heavier computation involved with WPF, the CPU footprint at the server does not need to scale significantly.

AJAX brings up another interesting development in the web world. Traditional web applications incorporated collections of loosely connected web pages whether server compiled JSP, ASP, PHP, or just plain old html. Web applications generally transmitted state from page to page often using server frameworks like Struts. In contrast AJAX makes monolithic web services feasible providing an alternative to the now familiar page based web applications. Many Ajax oriented sites of course still use some combination of stateless page progression and Ajax calls for specific page functions.

However, AJAX web services can more closely model the application model of development in which one executable controls the workflow, communicating with local computer resources such as databases, storage, and display. In Ajax resources spread across both the client and the cloud. This is important for the continued growth of software as a service, SaaS. Microsoft is apparently in the process of establishing a technology base from which SaaS can evolve. In essence Microsoft hopes to own the technology base undergirding the internet SaaS trend. It is easy to see parallels of this to OS development in the 80s, when Microsoft drove down to the software basement, ignoring hardware, and consequently owned the software superstructure by the end of the following decade.

An Ajax approach is able to send a monolithic web application controller to the client. Then, instead of moving from page to page as the user interacts with the browser, messages are pushed and pulled from the originating server modifying this single browser page. State can be stored in the client and complicated state transmission from page to page is avoided. So for example in Google Maps a single view continues to host user interactions while the view changes according to user actions. The page state progression with annoying refreshes is largely absent. This is all made possible with http calls shuttling back and forth from the monolithic client web page and Google’s server farms.

WPF makes sophisticated views much easier to develop and blurs the interface distinction often found between primitive web applications and traditional applications. Replace javascript, html, and bitmaps with C#, vector rendering, 3D models, and a full complement of interface controls and there are opportunities for much more sophisticated web application interfaces. Much of the power resides in the local rendering while the cloud provides the shared data and communications resources.

One web application I recently worked on illustrates WPF served from Apache Tomcat using AJAX calls to Java to provide a 2D GMTI viewer. This web app allowed user selectable GMTI data displayed on a map background. Once a mission is chosen the map view is adjusted with typical zoom and pan functions using matrix operations in WPF C# that take advantage of the client GPU. In addition the map view uses a variety of public OGC web map services as backgrounds. The selected background images are actually provided by OGC WMS sites and updated in the WPF client as the view changes with zoom and pan gestures. WPF partial trust Xbap applications are restricted to originating server resources. In order to take advantage of all the public image data behind WMS services on the internet cloud it is a simple matter to write a Java servlet proxy.This proxy servlet takes view parameters from the WPF client and posts them off in proper format to a WMS service. The resulting bitmap stream is then asynchronously loaded as a WPF image in the client view.

In this simple case the world of public map atlases becomes part of the application interface using a single xbap client and a Java proxy servlet. The resulting browser app runs more like a local application than a browser app but has the advantage of access to multiple world wide map and image services.

Once a map view is adjusted to the user’s satisfaction a tab change to ‘filters’ allows a user to set up different track sorting algorithms. Filter parameters are adjusted visually with WPF tools in the tab control. Once the parameters are set a filter button triggers an upload to the server where a filter grouping algorithm is triggered to go through the stored GMTI track points and identify tracks based on sequential spatial proximity and timestamps. Once the algorithm completes its pass through the selected GMTI mission, the user can choose to display the results in tabular or vector format.

AJAX is again handling the resource communication between client and server. Parameters from the interface flow to the server while results flow back to the client. However, in this case there are two entirely different worlds talking to each other, WPF C# on the client and Apache Java on the server. This magic is possible because of the decoupling provided by XML. XML parameter files are sent to java servlets which in turn respond with Xaml XML results to the C# client. This magic is greatly helped by two .NET classes XamlReader and XamlWriter. Using XamlWriter.Save on a WPF object serializes to an XML text that can be sent to the server. The return trip result can be picked up with the inverse XamlReader.Load to parse the response XML text into a WPF object.                          

StringReader stringReader = new StringReader(responseFromServer);
XmlReader xmlReader = XmlReader.Create(stringReader);
Canvas geom = (Canvas)XamlReader.Load(xmlReader);
canv.Children.Add(geom);
canv.RegisterName(geom.Name, geom);

The Xbap client does not care how the Xaml is created and the Java servlet does not care either, which lets two very distinct worlds talk to each other. This is very nice for the developer who can take advantage of high performance Microsoft WPF client interface controls and at the same time use a Linux based virtual servers as EC2 instances.


Fig 2 WPF pan zoom and scroll handled by local client C# and Imagery background from OWS services


Fig 3 XML bridges makes it possible to have Java filters feed to WPF geometries in the map view


Fig 4 Imagery backgrounds are interchangeable and vector geometries include event listeners for rollover and select