Creating Local Tile Servlet

I meant to write my updates when I got back from my trip a week ago, but there are a lot of things at home to take care of after a three week vacation. As promised I did some work on the WorldWind while I was away.

Creating Local Tile Server
With an unsteady internet connection I needed to be able to continue to test WorldWind changes that related to the download code. If I could set up a local server, and point my WorldWind program to check that for tiles I could still check the download code if it works. A side benefit is that in theory if my local tile server is something I coded myself, I'd be able to insert dummy error response codes like 404 or 500 errors in order to test the behavior for those errors in the new download code.

I tried using existing scripts and code that was posted on the WorldWindCentral wiki. But I either don't have the right application server software to run them (PHP, Perl,Python) or the know-how to set it up and use it (IIS 7/aspx.cs). There is also a WorldWind server, written in java posted on the World Wind forums (NASA World Wind WMS Server Config), I tried that too, this would allow me to set up a local server but because it was a self-contained application, but wouldn't let me achieve my second goal of inserting my own custom errors for testing. Even so, it has dependencies on all or some of WorldWind Java SDK. Now on my own, but understanding better what needs to get done, I thought "Why not just write this in a language I know, Java and set up Tomcat with a simple web app?"

Taking the .aspx.cs example, I ported that to a Java Servlet class.

package org.wwc.ammianus.tomcat.tileserver;

import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.text.DecimalFormat;
import java.util.HashMap;

import javax.servlet.ServletException;
import javax.servlet.ServletOutputStream;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

/**
*

This servlet handles requests from the World Wind.NET application and returns
* tile images or error codes as appropriate. This was a port to Java of a ASP.NET file posted
* at http://worldwindcentral.com/wiki/Data_serving_scripts

*
* @author ammianus
* @version 0.1
*
*
*/
public class TileRequestHandler extends HttpServlet {

/**
* Generated by Eclipse
*/
private static final long serialVersionUID = -6596187549520730077L;
/**
* settings Map which theoretically could be extended for any layer used in WW
* for each layer, use the <DataSetName></DataSetName> tag value from @Images.xml file
* create three map entries <DataSetName>, <DataSetName>_EXT, <DataSetName>_MIME
*/
private HashMap _settings;

private int _requestNumber;

/**
* @see HttpServlet#HttpServlet()
*/
public TileRequestHandler() {
super();
//use these setting which theoretically could be extended for any layer used in WW
//for each layer, use the tag value from @Images.xml file
//create three map entries , _EXT, _MIME
_settings = new HashMap();
_settings.put("geocover2000", "C:\\Users\\Public\\WorldWindCache\\Cache\\Earth\\Images\\NASA Landsat Imagery\\Geocover 2000");
_settings.put("geocover2000_EXT", ".jpg");
_settings.put("geocover2000_MIME", "image/jpeg");

_requestNumber = 0;
}

/**
* @see HttpServlet#doGet(HttpServletRequest request, HttpServletResponse response)
*/
protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
//increment counter
_requestNumber++;

//Un-comment this to test whether WW handles 503 return code with Retry-After header
//
//first request we will send an error with a retry-after header
//
/*if(_requestNumber == 1){
response.setStatus(503);
//this appears to be case insensitive for WorldWind, I just picked a random retry-after period
response.setHeader("retry-after", "38");
System.out.println(_requestNumber+":Response: 503, with retry-after: 38");
return;
}*/

try{


//column
int X = Integer.parseInt(request.getParameter("X").toString());
//row
int Y = Integer.parseInt(request.getParameter("Y").toString());
//level
int L = Integer.parseInt(request.getParameter("L").toString());
//DataSet name
String T = request.getParameter("T").toString(); //T = Dataset in WorldWind parlance


//Un-comment this to test whether WW handles 500 return code
//
//if(T.equals("geocover2000")){
// throw new Exception("Force return of 500 - Server Error");
//}

//Arguments are in this format URL sent from world wind
//e.g. http://localhost:8080/TomcatTileServer/TileRequestHandler?T=geocover2000&L=0&X=47&Y=21
String filePath = _settings.get(T);//WebConfigurationManager.AppSettings[T];

String fileExt = _settings.get(T + "_EXT");
if(L < 3){
DecimalFormat df = new DecimalFormat("0000");
filePath += "\\"+L + "\\" + df.format(Y) + "\\" + df.format(Y) + "_" + df.format(X) + fileExt;
}else{
filePath += "\\level"+(L+1)+fileExt;
}
//request.get
System.out.println(_requestNumber+":Requested File: "+filePath);
File file = new File(filePath);

//if file (image for requested tile) is not found on server, return 404
if (!file.exists())
{
response.sendError(HttpServletResponse.SC_NOT_FOUND, "Not Found");
}
else
{
//set content length to file size
response.setContentLength((int) file.length());
//get mime type for this layer
response.setContentType(_settings.get(T + "_MIME"));
//get the output stream for the response, this will have the file written to it
ServletOutputStream stream = response.getOutputStream();

//read the file contents and write them to the outputstream
FileInputStream fis = new FileInputStream(file);
byte[] bytes = new byte[1024];
int counter = 0;
while(counter < file.length()){
fis.read(bytes);
counter += 1024;
stream.write(bytes);
}
fis.close();
stream.flush();
stream.close();

//done with response
}
}catch(Exception e){
//uncaught Exception return 500 error
response.sendError(500,e.getMessage());
System.out.println(_requestNumber+":Response: send 500 error");
}

}

/**
* @see HttpServlet#doPost(HttpServletRequest request, HttpServletResponse response)
*/

protected void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
// TODO Auto-generated method stub
}

}
As you can see, not much there. In my WEB-INF/web.xml, I included the new servlet

<?xml version="1.0" encoding="UTF-8"?>
<web-app xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://java.sun.com/xml/ns/javaee" web="http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd" schemalocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd" id="WebApp_ID" version="2.5">
<display-name>TomcatTileServer</display-name>
<welcome-file-list>
<welcome-file>index.html</welcome-file>
<welcome-file>index.htm</welcome-file>
<welcome-file>index.jsp</welcome-file>
<welcome-file>default.html</welcome-file>
<welcome-file>default.htm</welcome-file>
<welcome-file>default.jsp</welcome-file>
</welcome-file-list>
<servlet>
<description></description>
<display-name>TileRequestHandler</display-name>
<servlet-name>TileRequestHandler</servlet-name>
<servlet-class>org.wwc.ammianus.tomcat.tileserver.TileRequestHandler</servlet-class>
</servlet>
<servlet-mapping>
<servlet-name>TileRequestHandler</servlet-name>
<url-pattern>/TileRequestHandler</url-pattern>
</servlet-mapping>
</web-app>

Now I was typically running this from my Eclipse IDE (Eclipse J2EE 3.4.1, Apache Tomcat 6), but I also could export the project in Eclipse as a war and just drop it into Tomcat's webapps folder.

To access this server there is the format that the URL would take:
http://localhost:8080/TomcatTileServer/TileRequestHandler?T=geocover2000&L=0&X=48&Y=21

To break this down:

• T = the name of the layer, or DataSet name ( tag value from @Images.xml)
  
• L = level, (0 - x) where 0 is the highest level
• X = column
• Y = row
  

From this the TileServer, based on its implementation and organization of the images, will try to look up locally a file that matches that. In my case it was something like C:\Users\Public\WorldWindCache\Cache\Earth\Images\NASA Landsat Imagery\Geocover 2000\0\0021\0021_0049.jpg".

Integrating Local Server with WorldWind
Thanks to WorldWind forums for giving me this information. Now to make WorldWind point to my local tile servlet instead of the default NASA WMS server, you need to edit the @Images.xml file in your WorldWind's Config\Earth directory. There is probably an @Images.tmp, copy this and rename it to @Images.xml.

Edit the file, and for the DataSetName your local server will handle, update the tag such as:

<ServerUrl>http://localhost:8080/TomcatTileServer/TileRequestHandler</ServerUrl>

And then restart World Wind you should be good to go

Vacation

On my way to South Korea! Will be spending my vacation there for the next three weeks.

Img Source: http://commons.wikimedia.org/wiki/File:How_to_make_s_korean_flag.jpg

On the flight I hope to set up my world wind with a local tile server for doing further development on download code. Maybe some more updates soon.

Java HashMap vs. C# Hashtable

One of the things I have found has been more frustrating while picking up C# .NET for the World Wind project is that some of the simple basic programming constructs that are automatic for me in Java don't seem to be directly parallel in C#. On top of that the online documentation at the MSDN site and other places is either not intuitive or I am looking in the wrong areas.

One of the basic data structures I depend on in Java are Maps. One of the simple classes that implement the Map interface in Java is java.util.HashMap.

Like other data structure objects in Java, there is no special syntax to interact with this object, it has methods like other objects where you pass in arguments and get return values.

The most basic usage of the HashMap generally goes like this:

//construct the new map
HashMap javaMap = new HashMap();
//enter a new key value pair
javaMap.put("key","value");
//to get the all values out of the map
Set keys = javaMap.keySet();
Iterator keyIter = keys.iterator();
while(keyIter.hasNext()){
String key = (String) keyIter.next();
String value = (String) javaMap.get(key);
System.out.println(key+" = "+value);
}

So it is pretty straightforward usage (in my mind), the put method "puts" data into the Map and the "get" method retrieves data from the Map given a key. Great

C# on the other hand uses a more elaborate syntax and combination of properties, keywords and methods. Not that this is more difficult but it took some time for me to figure out for example, that there was no equivalent to the "get" method in Hashtable, that you needed to use the "[]" syntax as if this were like an array

For the equivalent C# as the above, you can use the System.Collections.Hashtable class which implements the IDictionary interface:

//construct the Hashtable
Hashtable cSharpTable = new Hashtable();
//enter new key value pairs
cSharpTable.Add("key", "value");
cSharpTable.Add("hello", "world");
//for all Keys, a public property on Hashtable class, print out the key and value
foreach (String key in cSharpTable.Keys)
{
String value = (String) cSharpTable[key];
System.Console.WriteLine(key + " = " + value);
}

So in some ways the C# code is definitely more concise. But for a Java programmer it feels a bit strange, instead of 3 methods in Java(all with clear Javadoc descriptions of use, IDE auto-complete and tool tips, same syntax as all other method calls in Java etc..) for put(),get(),keySet(), C# requires you to use three different language features, a method, Add(), a property on the object Keys (why not just use a method?), and a [] operator. Granted in Visual Studio 2005 typing "cSharpTable." and you would see a list of the public methods and properties including the Add() and Keys, but you would only see the auto-assist tool-tip if you knew to type "cSharpTable[" and then it would give you the basic description of the syntax "object Hashtable[object key]".

I guess it is just something I'll have to learn.

I basically learned Java on the fly on my first job. I learned it fairly easily because once you got it that everything is just a method its pretty easy to find and locate what you need, plus the clear and standardized format of Javadoc based sites makes learning new classes or finding methods fairly simple. Try comparing the ease of use of Sun's Java API site with Microsoft's MSDN .NET Framework Reference(see link to HashMap and Hashtable classes above). Is it really an apples to oranges comparison? Or is the MSDN site just not very readable (for that one class there is no list of methods, just pages of examples in multiple languages.

References:
Holzner, Steven Microsoft Visual C# .NET 2003 Kick Start. Indianapolis: Sams Publishing, 2004. pp243-245

Contribution Update

I went back and did some further rework for the changes to the "Download code" I submitted to the forum earlier. I posted these on the forum as well (still do not have commit access to SVN). There were no clear directions so I took some suggestions from forum users and combined that with already existing functionality (that was in the code but not used due to a bug I think I fixed).

I spent this day making these updates. Basically there were 3 files updated.

Changes include:

• Now individual tiles will be put in a HashTable if they get 3 404 errors. They will be prevented from being readded to the download queue. Pressing 'F5' will clear this HashTable as will restarting the program.
• After 5 failed tiles download attempts (tiles reaching the status above) the current anti-hammer logic kicks in to prevent further requests to that server for that layer. The default timeout period is 2 minutes before it will try to download for this layer again. Pressing 'Pressing F5' will clear this counter, and will stop the timeout and restart the download queue for this layer.
• When the above condition is reached it will display a message in the top right saying "Problem connecting to server... Trying again in 2 minutes" when that layer is rendered after a timeout is reached. This stays visible for the first 20 seconds of the timeout then will be fade out...this logic was actually in there currently, but I was able to confirm that you can see it now (plus I extended the time it is visible from 15 to 20 seconds).

World Wind 1.4.1 Issues

I reported two more issues based on my work over the weekend. I did not get a whole lot of responses yet.

Issue 1:
I notice that when you uncheck "Starfield" in the Layer Manager, it doesn't actually remove the stars that are in the background.

Response: confirmed issue

Issue 2:
I have noticed this on my new development environment which is Vista (64-bit).

Sometimes, but not every time (can't seem to pinpoint an obvious cause or pattern), after I have run World Wind from VCE2005 and exit it from the File menu, then try to Run it in Debug again it will not start up. There is no Worldwind.exe process running and the problem remains if I just restart Visual Studio.


This was a juicy one, I had seen this before but thought it was some change to the code I had made caused this problem.

Thankfully I found that in WorldWind.cs it is actually checking for a Window handle named 'NASA World Wind', finding one and then returning after sending arguments for it.

I have had to comment out this return. Otherwise I need to log off of Windows and log back on to fix the problem.

The bug is that probably the Window is not being shut down completely.

// If World Wind is already running, pass any commandline
// arguments from this instance, and quit.
IntPtr handle = GetWWHandle();
if (!System.IntPtr.Zero.Equals(handle))
{


if(args.Length>0)
NativeMethods.SendArgs( handle, string.Join("\n",args) );

//[Ammianus] by commenting out this return I can get it to start every time.
//return;
}

Both bugs are being discussed in this stickied thread on the World Wind forums.

First contribution?

Technically I did not check-in any changes to SVN, but I did suggest a fix for World Wind on the forums.

I have been spending a good while looking into the "download code" issue. I have taken a step back and this past weekend I dove into the code using my new development machine.

By tweaking a couple of lines of code I managed to unlock behavior already built in to the download classes that would prevent multiple failed downloads from repeating for a particular layer (after 5 failures, stop downloads for a timeout period of 2 minutes). This was slightly different from the expressed requirement to flag specific tiles for not being found and not retrying those.

So on April 5, 2009, I went back to the World Wind forums and posted my update including the changes to the two class files to make the existing functionality work correctly. I even got some replies by a few active members that they would try it out. I asked for clarification as well if I should go back and make the anti-hammer logic work for tiles specifically rather than whole layers which is what I initially wanted to do.

It was such a successful feeling on Sunday to get this working the way I wanted. My wife and I spent all day working on our various projects, me on World Wind, her on her doctoral dissertation. It took about 5-6 hours of steady work, debugging and reading log files to really wrap my head around how these classes were truly interacting. Making a simple change (which I understood) and having it work was very rewarding, it gives me confidence to continue with some more changes.

Things to check:

* One recommendation was to display a message in the UI that the connection to the server is on hold for the 2 minute timeout period. I did in fact see something like that at one point in my testing but it quickly went away, may have been unrelated to my code changes
* Pressing 'F5' will clear the cache for a specific area and force a retry of the download of the visible tiles. Suggestion was that this should reset any timeout or counters of failures if the user doesn't want to wait (I suppose would be the case if they scrolled back over some location that they were more confident had tiles to download from the server)
* Need validation on whether current anti-hammer logic is ok, or we need per-tile anti-hammer logic

New development machine

Not wanting to abuse my work laptop too much, I decided that for my own personal development I needed a new machine, laptops are relatively cheap these days.

I decided on getting a lower-end machine (which means less on the CPU and Graphics card side, but leaving ample memory and hard disk space). After browsing around different reviews and shopping a bit, I decided to settle on a Toshiba Satellite M305-S4907

Basic specs:
* 2GHz Intel® Pentium® Processor T3400
* 4GB PC6400 DDR2 800MHz SDRAM
* 320GB HDD (5400rpm)
* Windows Vista® Home Premium (SP1, 64-bit)
* 14.1" widescreen display
* 5.2 lbs

It took me a few weekends to get used to how use Windows Vista, but I am really enjoying this pc now. I recommend it. Graphics scored somewhat low on the Windows Vista benchmark, but other than World Wind, I don't really use any graphics intensive applications. Unfortunately, World Wind is fairly graphics intensive and that was one of the primary reasons I bought this laptop, to do open source development when not at home.