Media Playing down the LiDAR path



Fig 1 – Showing a sequence of LiDAR Profiles as a video clip

Video codecs harnessed to Silverlight MediaPlayer make a useful compression technique for large image sets. The video, in essence, is a pointer into a large library of image frames. Data collected in a framewise spatial sense can leverage this technique. One example where this can be useful is in Mobile Asset Collection or MAC UIs. MAC corridors can create large collections of image assets in short order.

Another potential for this approach is to make use of conventional LiDAR point cloud profiles. In this project I wanted to step through a route, collecting profiles for later viewing. The video approach seemed feasible after watching cached profiles spin through a view panel connected to a MouseWheel event in another project. With a little bit of effort I was able to take a set of stepwise profiles, turn them into a video, and then connect the resulting video to a Silverlight Map Control client hooked up to a Silverlight MediaPlayer. This involved three steps:

1. Create a set of frames

Here is a code snippet used to follow a simple track down Colfax Ave in Denver, west to east. I used this to repeatedly grab LidarServer WMS GetProfileData requests, and save the resulting .png images to a subdirectory. The parameters were set to sweep at a 250ft offset either side of the track with a10ft depth and a 1 ft step interval. The result after a couple of hours was 19164 .png profiles at 300px X 500px.

The code basically starts down the supplied path and calculates the sweep profile endpoints at each step using the Perpendicular function. This sweep line supplies the extent parameters for a WMS GetProfileData request.

private void CreateImages(object sender, RoutedEventArgs e)
{
    string colorization = "Elevation";
    string filter = "All";
    string graticule = "True";
    string drapeline = "False";
    double profileWidth = 300;
    double profileHeight = 500;

    double dx = 0;
    double dy = 0;
    double len = 0;
    double t = 0;
    string profileUrl = null;
    WebClient client = new WebClient();

    step = 57.2957795 * (step * 0.3048 / 6378137.0); //approx dec degree
    Point startpt = new Point();
    Point endpt = new Point();
    string[] lines = points.Text.Split('\r');
    startpt.X = double.Parse(lines[0].Split(',')[0]);
    startpt.Y = double.Parse(lines[0].Split(',')[1]);

    endpt.X = double.Parse(lines[1].Split(',')[0]);
    endpt.Y = double.Parse(lines[1].Split(',')[1]);

    dx = endpt.X - startpt.X;
    dy = endpt.Y - startpt.Y;
    len = Math.Sqrt(dx * dx + dy * dy);

    Line direction = new Line();
    direction.X1 = startpt.X;
    direction.Y1 = startpt.Y;
    width *= 0.3048;
    int cnt = 0;
    t = step / len;

    while (t <= 1)
    {
        direction.X2 = startpt.X + dx * t;
        direction.Y2 = startpt.Y + dy * t;

        Point p0 = Perpendicular(direction, width / 2);
        Point p1 = Perpendicular(direction, -width / 2);

        p0 = Mercator(p0.X, p0.Y);
        p1 = Mercator(p1.X, p1.Y);

        profileUrl = "http://www.lidarserver.com/drcog?SERVICE=WMS&VERSION=1.3&REQUEST=GetProfileView"+
                          "&FORMAT=image%2Fpng&EXCEPTIONS=INIMAGE&CRS=EPSG:3785"+
                         "&LEFT_XY=" + p0.X + "%2C" + p0.Y + "&RIGHT_XY=" + p1.X + "%2C" + p1.Y +
                         "&DEPTH=" + depth + "&SHOW_DRAPELINE=" + drapeline + "&SHOW_GRATICULE=" + graticule +
                         "&COLORIZATION=" + colorization + "&FILTER=" + filter +
                         "&WIDTH=" + profileWidth + "&HEIGHT=" + profileHeight;

        byte[] bytes = client.DownloadData(new Uri(profileUrl));
        FileStream fs = File.Create(String.Format(workingDir+"img{0:00000}.png", cnt++));
        BinaryWriter bw = new BinaryWriter(fs);
        bw.Write(bytes);
        bw.Close();
        fs.Close();

        direction.X1 = direction.X2;
        direction.Y1 = direction.Y2;
        t += step / len;
    }
}

private Point Perpendicular(Line ctrline, double dist)
{
    Point pt = new Point();
    Point p1 = Mercator(ctrline.X1, ctrline.Y1);
    Point p2 = Mercator(ctrline.X2, ctrline.Y2);

    double dx = p2.X - p1.X;
    double dy = p2.Y - p1.Y;
    double len = Math.Sqrt(dx * dx + dy * dy);
    double e = dist * (dx / len);
    double f = dist * (dy / len);

    pt.X = p1.X - f;
    pt.Y = p1.Y + e;
    pt = InverseMercator(pt.X, pt.Y);
    return pt;
}

2. Merge the .png frames into an AVI

Here is a helpful C# AviFile library wrapper. Even though it is a little old, the functions I wanted in this wrapper worked just fine. The following WPF project simply takes a set of png files and adds them one at a time to an avi clip. Since I chose the (Full)uncompressed option, I had to break my files into smaller sets to keep from running into the 4Gb limit on my 32bit system. In the end I had 7 avi clips to cover the 19,164 png frames.


Fig 2 – Create an AVI clip from png frames

using System.Drawing;
using System.Windows;
using AviFile;

namespace CreateAVI
{

    public partial class Window1 : Window
    {
        public Window1()
        {
            InitializeComponent();
        }

        private void btnWrite_Click(object sender, RoutedEventArgs e)
        {
            int startframe = int.Parse(start.Text);
            int frameInterval = int.Parse(interval.Text);
            double frameRate = double.Parse(fps.Text);
            int endframe = 0;

            string currentDirName = inputDir.Text;
            string[] files = System.IO.Directory.GetFiles(currentDirName, "*.png");
            if (files.Length > (startframe + frameInterval)) endframe = startframe + frameInterval;
            else endframe = files.Length;
            Bitmap bmp = (Bitmap)System.Drawing.Image.FromFile(files[startframe]);
            AviManager aviManager = new AviManager(@currentDirName + outputFile.Text, false);
            VideoStream aviStream = aviManager.AddVideoStream(true, frameRate, bmp);

            Bitmap bitmap;
            int count = 0;
            for (int n = startframe+1; n < endframe; n++)
            {
                if (files[n].Trim().Length > 0)
                {
                    bitmap = (Bitmap)Bitmap.FromFile(files[n]);
                    aviStream.AddFrame(bitmap);
                    bitmap.Dispose();
                    count++;
                }
            }
            aviManager.Close();
        }
    }
}

Next I used Microsoft Expression Encoder 3 to encode the set of avi files into a Silverlight optimized VC-1 Broadband variable bitrate wmv output, which expects a broadband connection for an average 1632 Kbps download. The whole path sweep takes about 12.5 minutes to view and 53.5Mb of disk space. I used a 25fps frame rate when building the avi files. Since the sweep step is 1ft this works out to about a 17mph speed down my route.

3. Add the wmv in a MediaPlayer and connect to Silverlight Map Control.

MediaPlayer

I used a similar approach for connecting a route path to a video described in “Azure Video and the Silverlight Path”. Expression Encoder 3 comes with a set of Silverlight MediaPlayers templates. I used the simple “SL3Standard” template in this case, but you can get fancier if you want.

Looking in the Expression Templates subdirectory “C:\Program Files\Microsoft Expression\Encoder 3\Templates\en”, select the ExpressionMediaPlayer.MediaPlayer template you would like to use. All of the templates start with a generic.xaml MediaPlayer template. Add the .\Source\MediaPlayer\Themes\generic.xaml to your project. Then look through this xaml for <Style TargetType=”local:MediaPlayer”>. Once a key name is added, this plain generic style can be referenced by MediaPlayer in the MainPage.xaml
<Style x:Key=”MediaPlayerStyle” TargetType=”local:MediaPlayer”>

<ExpressionMediaPlayer:MediaPlayer
   x:Name=”VideoFile”
  Width=”432″ Height=”720″
  Style=”{StaticResource MediaPlayerStyle}”
/>

It is a bit more involved to add one of the more fancy templates. It requires creating another ResourceDictionary xaml file, adding the styling from the template Page.xaml and then adding both the generic and the new template as merged dictionaries:

<ResourceDictionary.MergedDictionaries>
  <ResourceDictionary Source=”generic.xaml”/>
  <ResourceDictionary Source=”BlackGlass.xaml”/>
</ResourceDictionary.MergedDictionaries>

Removing unnecessary controls, like volume controls, mute button, and misc controls, involves finding the control in the ResourceDictionary xaml and changing Visibility to Collapsed.

Loading Route to Map

The list of node points at each GetProfileData frame was captured in a text file in the first step. This file is added as an embedded resource that can be loaded at initialization. Since there are 19164 nodes, the MapPolyline is reduced using only modulus 25 nodes resulting in a more manageable 766 node MapPolyline. The full node list is still kept in a routeLocations Collection. Having the full node list available helps to synch with the video timer. This video is encoded at 25fps so I can relate any video time to a node index.

private List<location> routeLocations = new List<location>();

 private void LoadRoute()
 {
     MapPolyline rte = new MapPolyline();
     rte.Name = "route";
     rte.Stroke = new SolidColorBrush(Colors.Blue);
     rte.StrokeThickness = 10;
     rte.Opacity = 0.5;
     rte.Locations = new LocationCollection();

     Stream strm = Assembly.GetExecutingAssembly().GetManifestResourceStream("OnTerra_MACCorridor.corridor.direction.txt");

     string line;
     int cnt = 0;
     using (StreamReader reader = new StreamReader(strm))
     {
         while ((line = reader.ReadLine()) != null)
         {
             string[] values = line.Split(',');
             Location loc = new Location(double.Parse(values[0]), double.Parse(values[1]));
             routeLocations.Add(loc);
             if ((cnt++) % 25 == 0) rte.Locations.Add(loc);// add node every second
         }
     }
     profileLayer.Children.Add(rte);
 }

A Sweep MapPolyline is also added to the map with an event handler for MouseLeftButtonDown. The corresponding MouseMove and MouseLeftButtonUp events are added to the Map Control, which sets up a user drag capability. Every MouseMove event calls a FindNearestPoint(LL, routeLocations) function which returns a Location and updates the current routeIndex. This way the user sweep drag movements are locked to the route and the index is available to get the node point at the closest frame. This routeIndex is used to update the sweep profile end points to the new locations.

Synchronzing MediaPlayer and Sweep location

From the video perspective a DispatcherTimer polls the video MediaPlayer time position every 500ms. The MediaPlayer time position returned in seconds is multiplied by the frame rate of 25fps giving the routeLocations node index, which is used to update the sweep MapPolyline locations.

In reverse, a user drags and drops the sweep line at some point along the route. The MouseMove keeps the current routeIndex updated so that the mouse up event can change the sweep locations to its new location on the route. Also in this MouseLeftButtonUp event handler the video position is updated dividing the routeIndex by the frame rate.
VideoFile.Position = routeIndex/frameRate;

Summary

Since Silverlight handles media as well as maps it’s possible to take advantage of video codecs as a sort of compression technique. In this example, all of the large number of frame images collected from a LiDAR point cloud are readily available in a web mapping interface. Connecting the video timer with a node collection makes it relatively easy to keep map position and video synchronized. The video is a pointer into the large library of LiDAR profile image frames.

From a mapping perspective, this can be thought of as a raster organizational pattern, similar in a sense to tile pyramids. In this case, however, a single time axis is the pointer into a large image set, while with tile pyramids 3 axis pointers access the image library with zoom, x, and y. In either case visually interacting with a large image library enhances the human interface. My previous unsuccessful experiment with video pyramids attempted to combine a serial time axis with the three tile pyramid axis.I still believe this will be a reality sometime.

Of course there are other universes than our earth’s surface. It seems reasonable to think dynamic visualizer approaches could be extended to other large domains. Perhaps Bioinformatics could make use of tile pyramids and video codecs to explore Genomic or Proteomic topologies. It would be an interesting investigation.

Bioinformatics is a whole other world. While we are playing around in “mirror land” these guys are doing the “mirror us.”



Fig 1 – MediaPlayer using BlackGlass template

Map Clipping with Silverlight



Fig 1 – Clip Map Demo

Bing Maps Silverlight Control has a lot of power, power that is a lot of fun to play with even when not especially practical. This weekend I was presented with a challenge to find a way to show web maps, but restricted to a subset of states, a sub-region. I think the person making the request had more in mind the ability to cut out an arbitrary region and print it for reporting. However, I began to think why be restricted to just the one level of the pyramid. With all of this map data available we should be able to provide something as primitive as coupon clipping, but with a dynamic twist.

Silverlight affords a Clip masking mechanism and it should be simple to use.

1. Region boundary:

The first need is to compile the arbitrary regional boundary. This would be a challenge to code from scratch, but SQL Server spatial already has a function called “STUnion.” PostGIS has had an even more powerful Union function for some time, and Paul Ramsey has pointed out the power of fast cascaded unions. Since I’m interested in seeing how I can use SQL Serrver, though, I reverted to the first pass SQL Server approach. But, as I was looking at STUnion it was quickly apparent that this is a simple geo1.STUnion(geo2) function and what is needed is an aggregate union. The goal is to union more than just two geography elements at a time, preferably the result of an arbitrary query.

Fortunately there is a codeplex project, SQL Server Tools, which includes the very thing needed, along with some other interesting functions. GeographyAggregateUnion is the function I need, Project/Unproject and AffineTransform:: will have to await another day. This spatial tool kit consists of a dll and a register.sql script that is used to import the functions to an existing DB. Once in place the functions can be used like this:

SELECT dbo.GeographyUnionAggregate(wkb_geometry)) as Boundary
FROM [Census2009].[dbo].[states]
WHERE NAME = ‘Colorado’ OR NAME = ‘Utah’ or NAME = ‘Wyoming’

Ignoring my confusing choice of geography column name, “wkb_geometry,” this function takes a “geography” column result and provides the spatial union:

Or in my case:


Fig 3 – GeographyUnionAggregate result in SQL Server

Noting that CO, WY, and UT are fairly simple polygons but the result is 1092 nodes I tacked on a .Reduce() function.
dbo.GeographyUnionAggregate(wkb_geometry).Reduce(10) provides 538 points
dbo.GeographyUnionAggregate(wkb_geometry).Reduce(100) provides 94 points
dbo.GeographyUnionAggregate(wkb_geometry).Reduce(100) provides 19 points

Since I don’t need much resolution I went with the 19 points resulting from applying the Douglas-Peuker thinning with a tolerance factor of 1000.

2. Adding the boundary

The next step is adding this union boundary outlining my three states to my Silverlight Control. In Silverlight there are many ways to accomplish this, but by far the easiest is to leverage the builtin MapPolygon control and add it to a MapLayer inside the Map hierarchy:

<m:MapLayer>
  <m:MapPolygon x:Name=”region”
  Stroke=”Blue” StrokeThickness=”5″
    Locations=”37.0003960382868,-114.05060006067 37.000669,-112.540368
    36.997997,-110.47019 36.998906,-108.954404
         .
        .
        .
    41.996568,-112.173352 41.99372,-114.041723
     37.0003960382868,-114.05060006067 “/>
</m:MapLayer>


Now I have a map with a regional boundary for the combined states, CO, WY, and UT.

3. The third step is to do some clipping with the boundary:

UIElement.Clip is available for every UIElement, however, it is restricted to Geometry clipping elements. Since MapPolygon is not a geometry it must be converted to a geometry to be used as a clip element. Furthermore PathGeometry is very different from something as straight forward as MapPolygon, whose shape is defined by a simple LocationCollection of points.

PathGeometry in XAML:


<Canvas.Clip>
  <PathGeometry>
    <PathFigureCollection>
      <PathFigure StartPoint=”1,1″>
        <PathSegmentCollection>
          <LineSegment Point=”1,2″/>
          <LineSegment Point=”2,2″/>
          <LineSegment Point=”2,1″/>
          <LineSegment Point=”1,1″/>
        </PathSegmentCollection>
      </PathFigure>
    </PathFigureCollection>
  </PathGeometry>
</Canvas.Clip>


The easiest thing then is to take the region MapPolygon boundary and generate the necessary Clip PathGeometry in code behind:

  private void DrawClipFigure()
  {
    if (!(MainMap.Clip == null))
    {
     &nbspMainMap.ClearValue(Map.ClipProperty);
    }
    PathFigure clipPathFigure = new PathFigure();
    LocationCollection locs = region.Locations;
    PathSegmentCollection clipPathSegmentCollection = new PathSegmentCollection();
    bool start = true;
    foreach (Location loc in locs)
    {
      Point p = MainMap.LocationToViewportPoint(loc);
      if (start)
      {
       clipPathFigure.StartPoint = p;
       start = false;
     }
     else
     {
      LineSegment clipLineSegment = new LineSegment();
      clipLineSegment.Point = p;
      clipPathSegmentCollection.Add(clipLineSegment);
     }
    }
    clipPathFigure.Segments = clipPathSegmentCollection;
    PathFigureCollection clipPathFigureCollection = new PathFigureCollection();
    clipPathFigureCollection.Add(clipPathFigure);

    PathGeometry clipPathGeometry = new PathGeometry();
    clipPathGeometry.Figures = clipPathFigureCollection;
    MainMap.Clip = clipPathGeometry;
  }

This Clip PathGeometry can be applied to the m:Map named MainMap to mask the underlying Map. This is easily done with a Button Click event. But when navigating with pan and zoom, the clip PathGeometry is not automatically updated. It can be redrawn with each ViewChangeEnd:
private void MainMap_ViewChangeEnd(object sender, MapEventArgs e)
{
  if (MainMap != null)
  {
    if ((bool)ShowBoundary.IsChecked) DrawBoundary();
    if ((bool)ClipBoundary.IsChecked) DrawClipFigure();
  }
}


This will change the clip to match a new position, but only after the fact. The better way is to add the redraw clip to the ViewChangeOnFrame:

MainMap.ViewChangeOnFrame += new EventHandler<MapEventArgs>(MainMap_ViewChangeOnFrame);

private void MainMap_ViewChangeOnFrame(object sender, MapEventArgs e)
{
  if (MainMap != null)
  {
    if ((bool)ShowBoundary.IsChecked) DrawBoundary();
    if ((bool)ClipBoundary.IsChecked) DrawClipFigure();
  }
}


In spite of the constant clip redraw with each frame of the navigation animation, navigation is smooth and not appreciably degraded.

Summary:

Clipping a map is not terrifically useful, but it is supported with Silverlight Control and provides another tool in the webapp mapping arsenal. What is very useful, are the additional functions found in SQL Server Tools. Since SQL Server spatial is in the very first stage of its life, several useful functions are not found natively in this release. It is nice to have a superset of tools like GeographyAggregateUnion, Project/Unproject,
and AffineTransform::.

The more generalized approach would be to allow a user to click on the states he wishes to include in a region, and then have a SQL Server query produce the boundary for the clip action from the resulting state set. This wouldn’t be a difficult extension. If anyone thinks it would be useful, pass me an email and I’ll try a click select option.



Fig 4 – Clip Map Demo

Mirror Land and the Last Millimeter


Microsoft EMG Interface Patent

Patent application number: 20090326406

Well that was pretty quick. This went across the radar just this morning. See yesterday’s post Mirror Land and the Last Foot.

“Microsoft’s connecting EMG sensors to arm muscles and then detecting finger gestures based on the muscle movement picked up by those sensors” REF: Engadget

Looks like one part of the “Last Millimeter” is already patented. In a millmeter map we pick up objects and rotate them in mirror land. At least they don’t use drills with an EMG interface. My no fly threshold is any interface device requiring trepanning! It is interesting to see the biological UI beginning to stick its nose in the tent.


Technology has its limits

Mirror Land and the Last Foot


Fig 1 – Bing Maps Streetside

I know 2010 started yesterday but I slept in. I’m just a day late.

Even a day late perhaps it’s profitable to step back and muse over larger technology trends. I’ve worked through several technology tides in the past 35 years. I regretfully admit that I never successfully absorbed the “Gang of Four” Design Patterns. My penchant for the abstract is relatively low. I learn by doing concrete projects, and probably fall into the amateur programming category often dismissed by the “professional” programming cognoscenti. However, having lived through a bit of history already, I believe I can recognize an occasional technology trend without benefit of a Harvard degree or even a “Professional GIS certificate.”

What has been striking me of late is the growth of mirror realities. I’m not talking about bizarre multiverse theories popular in modern metaphysical cosmology, nor parallel universes of the many worlds quantum mechanics interpretation, or even virtual world phenoms such as Second Life or The Sims. I’m just looking at the mundane evolution of internet mapping.


Fig 2 – Google Maps Street View

One of my first mapping projects, back in the late 80′s, was converting the very sparse CIA world boundary file, WDBI, into an AutoCAD 3D Globe (WDBI came on a data tape reel). At the time it was novel enough, especially in the CAD world, to warrant a full color front cover of Cadence Magazine. I had lots of fun creating some simple AutoLisp scripts to spin the world view and add vector point and line features. I bring it up because at that point in history, prior to the big internet boom, mapping was a coarse affair at global scales. This was only a primitive wire frame, ethereal and transparent, yet even then quite beautiful, at least to map nerds.


Fig 3 – Antique AutoCAD Globe WDBI

Of course, at that time Scientists and GIS people were already playing with multi million dollar image aquisitions, but generally in fairly small areas. Landsat had been launched more than a decade earlier, but few people had the computing resources to play in that arena. Then too, US military was the main driving force with DARPA technology undreamed by the rest of us. A very large gap existed between Global and Local scales, at least for consumer masses. This access gap continued really until Keyhole’s aquisition by Google. There were regional initiatives like USGS DLG/DEM, Ordnance Survey, and Census TIGER. However, computer earth models were fragmented affairs, evolving relatively slowly down from satellite and up from aerial, until suddenly the entire gap was filled by Google and the repercussions are still very much evident.

Internet Map coverage is now both global and local, and everything in between, a mirror land. The full spectrum of coverage is complete. Or is it? A friend remarked recently that they feel like recent talk in mobile LiDAR echos earlier discussions of “Last Mile” when the Baby Bells and Cable Comms were competing for market share of internet connectivity. You can glimpse the same echo as Microsoft and Google jocky for market share of local street resolution, StreetView vs Streetside. The trend is from a global coarse model to a full scale local model, a trend now pushing out into the “Last Foot.” Alternate map models of the real world are diving into human dimension, feet and inches not miles, the detail of the street, my local personal world.

LiDAR contributes to this mirror land by adding a partial 3rd dimension to the flat photo world of street side capture. LiDAR backing can provide the swivel effects and the icon switching surface intelligence found in StreetView and Streetside. LiDAR capture is capable of much more, but internet UIs are still playing catchup in the 3rd dimension.

The question arises whether GIS or AEC will be the driver in this new human dimension “mirror land.” Traditionally AEC held the cards at feet and inches while GIS aerial platforms held sway in miles. MAC, Mobile Asset Collection, adds a middle way with inch level resolution capability available for miles.


Fig 4 – Video Synched to Map Route

Whoever, gets the dollars for capture of the last foot, in the end it all winds up inside an internet mirror land.

We are glimpsing a view of an alternate mirror reality that is not a Matrix sci-fi fantasy, but an ordinary part of internet connected life. Streetside and Street View push this mirror land down to the sidewalk.

On another vector, cell phone locations are adding the first primitive time dimension with life tracks now possible for millions. Realtime point location is a first step, but life track video stitched on the fly into photosynth streams lends credence to street side contingency.

The Location hype is really about linking those massive market demographic archives to a virtual world and then back connecting this information to a local personal world. As Sean Gillies in “Utopia or Dystopia” pointed out recently there are pros and cons. But, when have a few “cons” with axes ever really made a difference to the utopian future of technology?

With that thought in mind why not push a little on the future and look where the “Last Millimeter” takes us?
    BCI Brain Computer Interface
    Neuronal Prosthetics


Fig 5 – Brain Computer Interface

Eye tracking HUD (not housing and urban development exactly)


Fig 6- HUD phone?

I’m afraid the “Last Millimeter” is not a pretty thought, but at least an interesting one.

Summary

Just a few technology trends to keep an eye on. When they get out the drill for that last millimeter perhaps it’s time to pick up an ax or two.