Ridings overlay using PostGIS, Mapnik, and Mapstraction
July 4th, 2010
Last month I finally got around to adding a ridings overlay to the "Find your MP" tool for How'd They Vote. Here is an overview and source code for the tiles generated.
Taking the existing PostGIS ridings polygons in my database (in EPSG:4326 aka lat-long), I chose to use Mapnik to generate a set of tiles to present on top of a Google Maps base layer (conveniently presented through Mapstraction). For a primer on tile addressing and the relevant projections, see Coordinates, Tile Bounds, and Projection. Please note that the ridings polygons are based on the GeoGratis Federal Electoral Districts of Canada Shapefile.
After some experimentation, I decided on the following map definition: (ridings-simple.xml)
<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE Map>
<Map bgcolor="transparent" srs="+proj=merc +a=6378137 +b=6378137
+lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m
+nadgrids=@null +no_defs +over">
<Style name="ridings">
<Rule>
<LineSymbolizer>
<CssParameter name="stroke">#692032</CssParameter>
<CssParameter name="stroke-width">2</CssParameter>
</LineSymbolizer>
</Rule>
</Style>
<Layer name="fed" srs="+init=epsg:4326">
<StyleName>ridings</StyleName>
<Datasource>
<Parameter name="type">postgis</Parameter>
<Parameter name="host">localhost</Parameter>
<Parameter name="dbname">db</Parameter>
<Parameter name="user">user</Parameter>
<Parameter name="password">pw</Parameter>
<Parameter name="table">fed</Parameter>
<Parameter name="estimate_extent">true</Parameter>
</Datasource>
</Layer>
</Map>
My initial attempt included styling the whole riding polygon with the colour of the winning party, but it looked silly as one zoomed well into the riding itself (why does the whole map have a red tinge?). Wanting to keep it simple, I opted for rendering just the riding boundaries in "Elections Canada Maroon", #692032.
A code snippet from Open Street Map came in handy for generating the tiles in parallel, and for already doing all the math regarding tile bounds and naming. To suit my needs, the following modifications were made to generate_tiles.py:
112c112 < if bytes == 103: --- > if bytes == 116: 113a114 > os.remove(tile_uri) 119a121,129 > def has_parent(tile_dir, z, x, y): > pz = z-1 > px = x/2 > py = y/2 > pz_str = "%s" % pz > px_str = "%s" % px > py_str = "%s" % py > parent_uri = tile_dir + pz_str + '/' + px_str + '/' + py_str + '.png' > return os.path.isfile(parent_uri) 167c177,182 < queue.put(t) --- > if (minZoom == maxZoom and z > 3 and not has_parent(tile_dir, z, x, y)): > printLock.acquire() > print " skipping", zoom, str_x, str_y > printLock.release() > else: > queue.put(t)
- 103 was changed to 116 because of bgcolor="transparent"
- empty tiles are deleted immediately after creation
- has_parent function was added to determine if a tile exists one zoom level higher (and skip generating it if not)
My version of generate_tiles.py, with the above modifications is available here.
Once I started generating the tiles, I quickly gained an appreciation for the sheer number of tiles. Disk space was quickly consumed, and I was on a trajectory to consume a few terabytes by the time I hit zoom level 18 (although I ultimately only went to level 16, and for Canada only). With each tile being 256 by 256 pixels, the tile count (entire globe) climbs as follows:
| Zoom | X by Y | Tiles |
|---|---|---|
| 0 | 1 by 1 | 1 |
| 1 | 2 by 2 | 4 |
| 2 | 4 by 4 | 16 |
| 3 | 8 by 8 | 64 |
| 4 | 16 by 16 | 256 |
| 5 | 32 by 32 | 1,024 |
| 6 | 64 by 64 | 4,096 |
| 7 | 128 by 128 | 16,384 |
| 8 | 256 by 256 | 65,536 |
| 9 | 512 by 512 | 262,144 |
| 10 | 1,024 by 1,024 | 1,048,576 |
| 11 | 2,048 by 2,048 | 4,194,304 |
| 12 | 4,096 by 4,096 | 16,777,216 |
| 13 | 8,192 by 8,192 | 67,108,864 |
| 14 | 16,384 by 16,384 | 268,435,456 |
| 15 | 32,768 by 32,768 | 1,073,741,824 |
| 16 | 65,536 by 65,536 | 4,294,967,296 |
| 17 | 131,072 by 131,072 | 17,179,869,184 |
| 18 | 262,144 by 262,144 | 68,719,476,736 |
The final python script I used to generate the complete tileset was:
import generate_tiles map_file = "/path/to/ridings-simple.xml" tile_dir = "/path/to/tileset/" bbox = (-142.0, 40.0, -47.0, 90.0) generate_tiles.render_tiles(bbox, map_file, tile_dir, 0, 8, "Ridings") generate_tiles.render_tiles(bbox, map_file, tile_dir, 9, 9, "Ridings") generate_tiles.render_tiles(bbox, map_file, tile_dir, 10, 10, "Ridings") generate_tiles.render_tiles(bbox, map_file, tile_dir, 11, 11, "Ridings") generate_tiles.render_tiles(bbox, map_file, tile_dir, 12, 12, "Ridings") generate_tiles.render_tiles(bbox, map_file, tile_dir, 13, 13, "Ridings") generate_tiles.render_tiles(bbox, map_file, tile_dir, 14, 14, "Ridings") generate_tiles.render_tiles(bbox, map_file, tile_dir, 15, 15, "Ridings") generate_tiles.render_tiles(bbox, map_file, tile_dir, 16, 16, "Ridings")
This narrowed the bounding box to Canada, generated all tiles for zoom levels 0 through 8, and then generated tiles having a parent tile in the tile pyramid (one zoom level at a time).
Finally, via Mapstraction v1, the following was added to enable the tile overlay:
mapstraction.addTileLayer('http://howdtheyvote.ca/tileset/{Z}/{X}/{Y}.png', 0.5, 'Elections Canada', 0, 16);
(This will be a bit more verbose when done directly with the Google Maps API, but is left as an exercise for the reader)
The Result
2.1G of PNG tiles generated over roughly 3 days (for zoom levels 0 through 16). When compressed, the tiles are a mere 113M.Areas for improvement
- optimize PNG files (when only one colour is used, images should be smaller)
- speed up processing time by splitting up the work (onto multiple EC2 instances?)
- prevent 404 errors using redirects to a default transparent PNG (or generate zero byte files)
- consider more attractive styles, adding labels, and possibly replacing the base layer
eBay Developers Conference 2008
June 20th, 2008
Last week I was off in Chicago to take in the eBay Developers Conference and a few hours of eBay Live! (and a wee bit of YAPC too).
The conference was a bit less technical than I had hoped, but the venue was excellent (brand new LEED-certified building), the food was fantastic, and I made a lot of great contacts within eBay to help keep our integration on track. I was very pleased to see that nearly all the conference slides were made available under a creative commons license (although strangely some eBay Live! sessions were completely locked down and no cameras were allowed).
We ended up staying in the Seneca Hotel, which is right next door to the John Hancock Centre. I had a pretty nice 15-minute walk down the magnificent mile each morning to catch the shuttle to the conference centre.
The amount of swag taken home from this conference was surprising; fortunately, the stylish conference bag was put to good use toting all that stuff home. The highlights were definitely the PayPal Slinkee and eBay universal power adapter. Worse swag ever goes to the Amazing Race DVD Board Game
I was a little put off by the flagrant self-promotion of salesforce.com, and the general irrelevance of some of the keynote speeches. Mind you, I ended up taking the 266-page salesforce platform guide home with me (which i'll likely read in a few months when i've forgotten about a certain keynote) -- so in the end, having 300 developers as a captive audience was a good investment. However, i'd still expect any speaker to be able to answer, without hesitation, "what the hell does this have to do with eBay?". There is of course, a bit of this at every conference one goes to, but most speakers are slightly more subtle.
Since a co-worker was a few blocks away at YAPC::NA 2008, I took in a little bit of that conference too. On Tuesday night I joined the "Conference Dinner and Auction", which was rather entertaining with Uri (the fat guy in a tye-dye t-shirt) making random, periodic, profane announcements into the bowling alley/pool hall/bar microphone; not for a second, was there any possibility of mixing up the perl and eBay conferences. On Wednesday afternoon, I snuck in (with permission) to the YAPC lightning talks and closing keynotes; a couple guys from Google (who i'd seen speak before) gave their canned "users" talk. The lightning talks were great as lightning talks usually are -- happy little brain dumps on things you may or may not care about -- and always entertaining.
It was an eventful week, and I picked up lots of random nuggets of information -- now I just need to be locked in a room for 3 weeks to work through the TODO list this week has created. For a random smattering of photos, see ebaydevcon08 and yapcna2008 on flickr.
2007 FOSS4G Conference
September 29th, 2007
This week I took a few days off to partake in the 2007 Free and Open Source Software for Geospatial Conference (FOSS4G2007) conference. Being a very mobile conference, I was quite fortunate to have it here in Victoria -- I simply couldn't pass it up (last year in Switzerland, next year in South Africa). Given that this is a hobby at the moment, I chose to volunteer; in retrospect, I regret this decision as I missed a number of presentations -- on the bright side, I did sit through some presentations which I would not have otherwise attended.
Rather than hypnotizing you with a well-crafted thesis, i'll leave you with various thoughts which were more representative of my experience -- inspired but convoluted.
Proj4js really caught me by surprise -- holy crap this is *fast*, but it should be given that it only has a few EPSG codes supported. In the web mapping sphere, there is some healthy competition; OpenLayers and MapGuide both have respective wrappers for ease of use: CartoWeb4 and Fusion. Both are projects I haven't had a chance to check out, but will. Fun public projects: SF Urban Forest, Victoria Green Map, Digitized Historical Maps.
Quotables:
Jo Walsh - "Once you realize you need an SDI, you don't need one any more".
Tim Bowden - "GIS is dead".
Mark Sondheim - "The OGC still has a little life left in it".
Random thought: Where the hell is Dave Blasby?
Thursday afternoon I sat down with a couple of Open Street Map proponents; my curiosities here were to figure out if this wealth of free data was useful and how I might become involved (or at the very least, to see if I could learn anything to pass on to the CivicAccess folks). I was startled to learn that these guys don't believe in spatial databases -- we'll see how things go as feature density increases. I'll also have to see how well one can extract OSM data into a local cache for manipulation.
Things to revisit: Geobase
Things to check out/play with: GeoKettle, OpenAerialMap, OSM, pgRouting, R (stats are fun)
Ideas:
- use pgRouting to perform routing for pedestrians and cyclists
- a geocoder in Canada one can actually *use*?
To-do list
- Hook-up ISF hotspots to HTV
- Research public geodata
- uDig as an Open Street Map editor
- SLD 1.1
[Pictures]
Victoria Rodent Data
July 29th, 2007
As a fun side project, i've been tinkering with the VIHA health inspections data for Victoria.
Essentially it's just roughly geocoded points based on an aggregated screen-scraping -- but it is fun to see which restaurants you've been to in the last while have had a rodent-related infraction.
Data is available here, in shapefile format. Above is a quick picture of all the places which deal with food in Victoria in green, and offenders in orange.
New Job!
July 18th, 2007
Yesterday was my last day at Refractions. I had some enjoyable projects and some not-so-enjoyable projects there, but it was a valuable use of the last two years.
Next week I start working at Abe Books, in the FillZ division. I've heard only good things about the company, and i'm looking forward to broadening my horizons. Sadly, I won't be part of Refractions when the FOSS 4G conference comes to town in September, but I do still plan to attend.
More Entries