In this white paper we will examine the performance of MapLarge API tile servers under real world load scenarios. We will also quantify some typical consumer use cases and estimate server requirements for a typical multi user load.
This article will explain how to deliver quick, responsive maps. Before we begin that discussion, we would like to briefly mention why improving the response times experienced by users is important. The chart below illustrates the costs of a single slow response time. Imagine the compounding frustration users experience as they struggle to accomplish a map task that requires numerous slow responses. For more detail see our detailed article on user experience.
Using browser based tools like Firebug for Mozilla Firefox, measuring the initial page load of a web map is straightforward. Below is a comparison of a simple Google map from maps.google.com on the left and a multi layer education mashup from FCATMap.com which is powered by the MapLarge API. The figure below shows the Google Map loading one set of tiles and FcatMap loading two sets of tiles: dynamic layer of Map Large API tiles, and a base map of Google street tiles. This additional complexity is typical of mashups that sit on top of a street map base layer.
Analysis of the usage patterns of consumer maps hosted on the Map Large API shows that the typical user engages in bursts of panning, zooming and clicking activity. When first engaging with the map, users generate bursts of activity as they pan and zoom to an area of interest and several smaller bursts as they explore a more detailed area and click on items of interest. The level of consumer engagement will vary by use case and the best way to measure is to watch the load generated by users actually engaging with the map. The figure below depicts four example use cases and the http requests involved. In each request the initial page was loaded first and the request log cleared so that the FireBug logs below show the additional http requests created as a result of user interaction with the map. Google Maps and MapCancer made only map related requests and showed fewer requests, and Zillow and Hotel Map Search made additional requests to load content around the map like images of homes or hotels. The average use case session lasted 30 seconds to a minute and involved 300 to 400 requests or an average of about 10 requests per second during an active user session.
Delivering static content under load is a fairly well understood problem. Most large sites use Content Delivery Networks (CDN) to offload larger static files from their dynamic servers and their dynamic servers also cache content in memory or on disk. This system works well for caching the files needed for the initial load of a map because every user of the page requires the same file and the system can simply cache the files in memory or on a CDN. Caching starts to become problematic as web 2.0 applications allow users to dig into deeper datasets and begin to generate dynamic content queries. Caching every permutation of every unique query could easily swamp the memory and file system limitations of most dynamic and CDN servers. Complex interim solutions are possible, but the ideal solution would be to have dynamic servers fast enough to handle millions of unique queries on the fly.
Delivering fast, interactive maps of large datasets to users is the challenge the benchmarks below address. The dataset used for the test below was a 58mb database of over 140,000 hotels with attributes covering price and other metadata. The icons were dynamically shaded by price using the attribute field "LowRate". Our Web Mercator tile format is compatible with all major base map vendors including Bing, Google and ArcGIS. All info about the map is conveyed in the url and you can experiment with different values here
In the test below 25 external http clients hit a single core Azure instance as rapidly as possible. The test uses random unique tile requests so that every tile was dynamically generated and all requests are focused in the zones where data is the thickest. All work is preformed by the single instance: There is no database server and no tiles are pre-cached. The instance ran under a sustained load for 3 hours and achieved constant throughput of 100 to 150 requests for every sample during the entire period. Below is a chart an example sample period.
We selected the smallest Microsoft Azure instance for these tests because they represent a nice baseline metric that can be easily monitored and scaled. The specs are as follows below.
For this test we monitored and logged CPU, Memory, Http throughput and server errors during a three hour run. Memory usage stayed constant, there were no errors. CPU usage was the bottle neck and http throughput was over 100 requests per second as shown below.
In the test below 100 external http clients hit a single core Azure instance as rapidly as possible. All requests were for a tile in cache. However, each http request included a unique extra variable in their query string to force a cache analysis and full client-to-server http round trip (no interim network or CDN caching).
This test also used a small Azure Instance (see specs above). A CDN or load balancer could also fill the role of handling cache responses in some circumstances.
As in the sustained test above, memory stayed constant and there were no errors. CPU appeared to be the bottleneck, and throughput was over 780 requests per second.
As we saw above, load profiles can vary considerably from application to application, but a typical consumer map load will generate in the neighborhood of 10 requests per second during active sessions when the user is engaged with the map to complete a task. The metrics above indicate that a single core Azure instance can support over 100 dynamic requests per second and over 700 cached requests per second. Dividing that by 10 requests per second per user that means a small Azure instance can support up to 10 (totally dynamic) and 70 (totally cached) active sessions without the support of a CDN or other load balancer. How many requests are served from cache vs. dynamically will vary with usage patterns but it is common in consumer applications to see more than 50% of requests served from cache. The system scales linearly across processors in the Azure instance family and duplicate instances can be cloned and deployed automatically or manually with a few clicks.