Archive for the ‘Mapping’ Category

Love #maps and #datavisualization? Please join me at NACIS this year in Greenville!

Thursday, April 11th, 2013

[Editor’s note: The call for proposals is now open! As NACIS vice president and program chair for this year’s conference, I cordially invite you to attend and please present.]

web-homepage-image


NACIS 2013

Greenville, SC
October 9-11, 2013
Hyatt Regency Greenville

CFP2013

Talks, Posters, Demonstrations, Round Tables, Panels, and Workshops

There are many ways you may participate! You are invited to submit proposals to organize custom sessions, panels, discussions, and workshops. Talks, papers, posters, and demonstration proposals are welcome until June 15th. We will entertain all creative ideas!

ABOUT NACIS

Our enduring passion is to render our social, economic, political and natural world intelligible and actionable through visual story and cartographic design.

We are a creative design-oriented group that for over thirty years has provided a forum spanning the diversity of the mapmaking, geospatial & data visualization communities. We are specialists from government, commercial, and not-for-profit fronts along with academics, scholars, map/GIS librarians, artists, and students. Be there for the dynamic discussions rich in techniques, theory, history, and the state of our art. Please join us!

CARTOGRAPHY AND GEOGRAPHIC DATA

This year’s theme is data and data-driven map stories. Cartographers rely more than ever on technology and data science to craft engaging and informative narratives. As geographic data gets bigger, how can we find and follow the story arc? How can we use technology to make the humble spreadsheet or personal narrative interactive and read just as well online as in print?

SPECIAL PROGRAMS

In addition to the main Thursday and Friday conference, we offer several special programs.

  • Practical Cartography Day  Join professional cartographers and scholars for this popular all-day session. Come prepared to share in their insightful solutions to everyday cartographic conundrums.Mamata Akella and Andy Woodruff pcd@nacis.org
  • Cartographic and Geographic Data Collections  This all-day session will focus on innovative ways cartographic and geographic data collections are being managed and disseminated. Abraham Kaleo Parrish and Tsering Wangyal Shawa pmld@nacis.org
  • Exhibits Gallery - Display your latest creation for all to ponder throughout the events. Daniel Huffman posters@nacis.org

Read more at NACIS »

Natural Earth v2.0.0 is live

Thursday, November 8th, 2012

I am proud to announce the immediate availability of Natural Earth 2.0.0!

The 2.0.0 release focuses on 7 major areas and is available to download today à la carte at NaturalEarthData.
ZIP combo downloads of all vectors: SHP (279 mb) or SQLite (222 mb) or QuickStart kit for ArcMap and QGIS (165 mb).
  1. Economic geography: adds global roads, railroads, ports, airports, and time zones to show how people are interconnected and goods route (read Richard Florida on airports, full legal document about time zones and international date line shifts, and background on the E-Road network).
  2. Remastered geometries: fixes topological errors at 1:10 to 1:1,000 scales in the basic coastline, ocean, land, admin-0, and admin-1 related themes for files in the the 1:10m scaleset. By removing self-intersections, sliver polygons, and adjusting offset polygons, Natural Earth imports into more GIS software (like PostGIS) and will be easier to maintain. The coastline is adjusted to better conform to ~1:3,000,000 satellite imagery. Because of all these changes, some raster themes are also updated. Land, ocean, and minor islands all build topologically by scripting ingredients, as do the admin-0 and admin-1 cultural themes.
  3. Introduce Gray Earth rasters. Worldwide terrain depicted monochromatically in shades of gray. It combines shaded relief and regionally adjusted hypsography that emphasizes both high mountains and the micro terrain found in lowlands. View new raster »
  4. New file name and field name schemas. Full adoption of ne_10m_theme_name.shp file names with `ne_` prefix to allow better import into GeoDB and PostGIS storage, lowercase field (column) names instead of MiXeD and UPPER cased names, and use of consistent `name` field (versus name1).
  5. Address user submitted bug reports, ~25 since the 1.4 release, and earlier.
  6. Moved to Github for the backend versioned file management and coordination. Includes scripts to package updates and auto-create derived themes. View Natural Earth Vector on Github »
  7. Adopt semantic versioning. Know, by theme, the level of effort needed to update your maps when Natural Earth data updates are released. Read more about Natural Earth versioning »

Other notable changes:

All themes now include README and VERSION files. The admin-0 attributes have more veracity and now includes nested disputed areas (was a sidecar). Adds continent, region, subregion codes. Adds versions of country and admin-1 without boundary lakes.  All places and parts of places have population and GDP estimates. The populated places pop_max and pop_min attributes are now fully built out for all records (pop max is for the metropolitan area, pop_min is for the incorporated city of the same name). populated places now include rank_max and rank_min for simple town size grading. All instances of name1 have been changed to name, name to name, name2 to name_alt. Vertexes were added to many themes to allow them to project into conics smoothly (they’re back!). All field (column) names are now generally in the order of: scalerank, featurecla, name, name_alt, natscale, labelrank, *.

Contributors

Many thanks to the individuals who contributed over the last year of development: Tom, Nathaniel, Alex Tait, Hans van der Maarel, Scott Zillmer, Mike Migurski, Daniel Huffman, Xan Gregg, Peter Bispham, Drew Noakes, Miguel Angel Vilela, Matthew Toro, Kevin Pickell, Shawn Allen, Robert Coup, Iain, Leo, and more! Thanks also to Stamen thru the Knight Foundation Citytracking grant for sponsoring a portion of this work including remastering geometries for better PostGIS import, the move to Github, and adopting semantic versioning.

Changelog

Over 225 files have been updated in Natural Earth 2.0.0. Abbreviated listing below.
  • UPDATED: NE_ADMIN_0 - Updated for South Sudan, map colors (now with 7, 8, 9 and 13 options), population figures, removed () from notes, and more. note: diffs between sov, adm0, map units, map subunits, and new breakaways are all calculated on the a3 codes now, no longer mix of names and a3 codes. Added and split note_adm0 and note_brk to note which countries are parts of which sovereignties and who’s breaking away or disputing. One spurious “county” feature code fixed to “country” (finland). Added labelrank on all. Added new mapcolors (7, 8, 9 and old 13). Includes new detail on Caribbean Netherlands map unit. Adds more detail to Bhutan disputed areas. Now includes continent codes, and future region code placeholder columns. Added name_len to know when to abbreviate labels. Added label ranks.
  • UPDATED: NE_10M_ADMIN_0_BOUNDARY_LINES_LAND - Minor updates to alignment of boundary lines (and topology fixes), additional coding to allow official US gov’t view of same. better disputed coding, including Kosovo. Densified vertex along lines to allow smooth projection into conics. Moved Omani exclave Madha to correct location. Adds left and right labels and codes. Fixes: N96NSYPAPV, ZQNTN5VGDD, Z8ZYYUQZVS.
  • UPDATED: NE_50M_ADMIN_1_STATES_PROVINCES_SHP – Added some new ISO coding, other minor changes. Fixes topology errors. Adds admin-1 for brazil and australia. Uses same coding as 10m files. Derived from new scale rank version.
  • **NEW**: NE_10M_ADMIN_0_ANTARCTICA_CLAIMS – Although countries have paused their claims to the southernmost continent, they haven’t suspended them. Thanks, Hans!
  • **NEW**: NE_10M_ADMIN_0_ANTARCTICA_CLAIM_LIMIT_LINES – Although countries have paused their claims to the southernmost continent, they haven’t suspended them. Thanks, Hans!
  • UPDATED: NE_10M_POPULATED_PLACES – A couple name corrections (Morelia, Mexico spelling fixed. Mazatlan, Mexico spelling fixed. Clarified confusion around Tabatinga / Leticia on the Colombian / Brazilian border. On the Brazil / Bolivia border, clarified Brasileia / Cobija. Fixed spelling of Shuozhou, China), many population max values, mostly in China, India, rift valley (Africa), Nigeria, and other countries in east Asia, but some elsewhere. Made sure cities in Switzerland are coded admin-0 of CH and China are CN. Moved Amundsen Base to 176° so it’s in the -12 timezone. Also moved Peter I Island. Vatican City is also moved to be contained by it’s admin-0 polygon. Same for San Marino. Added a poprank column with 0 to 14 numerical classes. Deleted spurious Extra Eureka town in Canada near Greenland. Delete duplicate town Urengoy in RUS, rename the real one Novy Urengoy. Fixes: 4SUAZ7BB49, D459XT1Z6Y.
  • UPDATED: NE_10M_COASTLINE – Better matches modern satellite imagery to zoom 8-ish. The earlier coastline could have been several kilometers off (like in Gibraltar). Several large new islands added. Includes densified vertex along lines to allow smooth projection into conics.
  • UPDATED: NE_10M_RIVERS_LAKE_CENTERLINES – See changelog for ne_10m_rivers_lake_centerlines_scale_ranks for details.
  • UPDATED: NE_10M_RIVERS_LAKE_CENTERLINES_SCALE_RANKS – Updated river names, few new rivers, splits. added river connector in Sweden between lake near Stockholm and Baltic Sea. Fixes in France and Netherlands. Fixes Mackenzie river at it’s confluence with Dawson river in Australia. Names the Mahakam in Borneo (Rivernum 544). Changes scalerank on Nelson river in Canada. Fixes: SHAWNZQJ3B, 5J47B13PJ7, W9X539LBUT, 35YLBL2W9Z.
  • UPDATED: NE_10M_RIVERS_LAKE_CENTERLINES_NORTH_AMERICA_SUPPLEMENT – Updated river names, few new rivers, splits. Fixes: SHAWNZQJ3B, 5J47B13PJ7.
  • UPDATED: NE_10M_RIVERS_LAKE_CENTERLINES_EUROPE_SUPPLEMENT – Updated river names, few new rivers, splits. fixed topology errors. Fixes: SHAWNZQJ3B.
  • UPDATED: NE_10M_LAKES – Removed major lake groupings (Great Lakes, Finger Lakes, etc) to geography label areas instead. Title cased the feature class values. Added Swedish lake near Stockholm (had been extension of Baltic Sea in ocean theme). Fixed topology errors. Fixed a few reservoir and salt lake codes (thanks Craig!).
  • UPDATED: NE_10M_LAKES_NORTH_AMERICA_SUPPLEMENT – Name1 have been changed to name, name to name, name2 to name_alt. Fixes 4VA9P9UGQE.
  • UPDATED: NE_10M_GEOGRAPHIC_LINES – New int’l date line, thanks Alex! Also densified linework for smoother projection into conics.
  • UPDATED: NE_10M_LAND – A dissolved version of the original 1.x file, now renamed “ne_10m_land_scale_rank”, see that changelog for full details. Fixes XAWXTN54GT.
  • **NEW**: NE_10M_LAND_SCALE_RANKS – Renamed our original land file to this. Incorporates new coastline. Includes densified vertex along lines to allow smooth projection into conics. Fixes XAWXTN54GT.
  • UPDATED: NE_10M_OCEAN – A dissolved version of the original 1.x file, now renamed “ne_10m_ocean_scale_rank”, see that changelog for full details. Fixes XAWXTN54GT.
  • **NEW**: NE_10M_OCEAN_SCALE_RANKS – Renamed our original ocean file to this. Incorporates new coastline. Removed Swedish lake near Stockholm (had been extension of Baltic Sea in ocean theme) to lakes layer. Incorporates new coastline. Includes densified vertex along lines to allow smooth projection into conics. Fixes XAWXTN54GT.

Announcing Natural Earth v2.0.0, release candidate 3

Tuesday, October 16th, 2012

Just in time for NACIS 2012 in Portland, Oregon!

This is a followup to the earlier version 2.0.0 release candidate 2. Read more about RC2 »

Downloads:

Please submit bugs or other oddness found release candidate 3 to myself nathaniel@kelsocartography.com ASAP. The plan is to cut a final 2.0.0 release by the end of October. View full combo change log for 2.0.0 »

What’s new in 2.0.0 release candidate 3:
  1. Added explicit labelrank in the admin-0 XLS doc. Also added homepart (map units, map subunits, brkaway) and tiny (country) to better deal with labeling awkwardness and adds consistency between themes and scalesets.
  2. Updated sr_label_i, sr_label_o  on tiny_countries for better labeling.
  3. Added 3 new “mapcolors” for 7-up, 8-up, and 9-up into details doc and onto the admin-0 and admin-1 themes. This adds to the 13-up original. The 9-up is optimal, the others are useful in their own ways. 
  4. Renamed the “note” in the admin-0 files to “note_adm0” and added details for Korea and Japan parts (just the exterior bits). Added AUZ note for CSI, Cayman Islands, Turks, etc.
  5. Added the disputed areas note directly to the admin-0 details, named “note_brk“.
  6. Fixed admin-0 coding with Georgia to fix map units bug. Added new Caribbean Netherlands and metropolitan Netherlands codes to reflect rejigger there.
  7. Added continent codes for admin-0 countries and their various parts. Stubbed out region and subregion fields, values to come later release. Thanks SmugMug!
  8. Removed “County” admin-0 feature class, remapping it to “Country” (Finland only).
  9. Removed parenthetical () from notes, see Faulkland Is. etc. Removed all claim notes except from the BRK bits in admin-0 details. Add that while labeling now (consistent with other app logic now).
  10. Add admin-0 diffs for sov, adm0, map units, map subunits, and disputed areas are now calculated on the a3 code, not the name. This can be used for progressive labeling, fixing problems in Tanzania and elsewhere. You still need logic in your app to determine if the labeling name is different, but now it’s consistent.
  11. Added Taiwan to disputed areas. Renamed Congos (eg Democratic Republic of Congo instead of Congo (Kinshasa)).
  12. Modified the scalerank of Venezuala claim into Guyana boundary line fix 10m theme. Similar for the Saichen Glacier.
  13. Changed the scaleranks on all 10m claim polys. Most now appear at scalerank 6 and in.
  14. For labeling, added name_len to the admin 1 and admin 0 details and themes.
  15. Changed several scaleranks of marine label areas in 10m and 50m themes.
  16. Added continent code for roads (partial) so North America and Europe can be separated out easier. Might build topology for roads in final release to make future cleanup easier.
  17. Updated, added new Airports by rerunning export from Mile High Club, with better field names.
  18. Updated, added new Ports by rerunning export from High Seas, with better field names. This added inland ports especially along the North American Great Lakes.
  19. Promoted a few populated places up to scalerank 1 and 2 to give better balance to compilation. Many cities have more accurate pop_max values (partial, more due in final release). Thanks for the nudge Craig!
  20. National park (protected areas) points were missing “name”. Some reranking for park points in urban centers (to scaleranks 7, 8, 9). A couple park areas reranked around scalerank 3. Added Cesar Chavez NM and added missing unit_codes for Clinton Birthplace NHS and River Raisin NBP. Thanks Tom!
  21. Add admin_0 alpha3 codes to boundary lines to allow better selection (grab the boundaries of USA only, for instance). Also added left and right names to allow better admin-0 line labeling on zoom in.
  22. Parts of Chile and Cyprus were missing in derived admin-0 themes (were present in scale rank masters) so modified some label points so their scalerank is now <= 6 for admin-0 selection. These had been 7 and 8.
  23. Removed old versions of admin-1 lines from the ZIPs.
  24. Added left and right names for admin_1 boundary lines for better labeling once zoomed in. Because native names are in UTF8 (from the polygons), also added a “master” lines version in GeoDB format along with the SHP version, sidestepping character encoding problems. Thanks Mamata and Mike!
  25. The admin-1 boundary between Tasmania and mainland Australia is now straight (thanks Craig!). Adjusted admin-1 boundaries in Bhutan to match new admin-0 disputed areas and boundaries.
  26. Added Australia and Brazil admin-1 lines and polys to 50m. Redid the 50m and 110m admin-1 attributes (lines, polys) to match the 10m. Added scale_rank version of the polys at 50m and 110m to allow for easier updates. Recut a lakes erase version of the 50m admin-1, but now using lakes with scalerank <= 0 instead of 1 (North America’s Great Lakes, but not Florida or Utah lakes) at Mike Bostock’s suggestion.
  27. Modified scaleranks on the 10m urban areas and copied those over to the 50m urban areas. Deleted a bit of 10m urban on Mt. Hood, Oregon snowfield.
  28. Redid marine label area scale ranks.
  29. Removed duplicate Laccadive Is. from 10m-geography-regions.
  30. Fixed river scalerank bug in 10m theme in Canada for the Nelson River leading out of Lake Winnipeg, thanks Gregor!

Still to do for final release:

  • Rebuild the 10m, 50m, and 110m admin 0 themes using the new details. Add area_sqkm to all themes after build.
  • Merge in modified 10m China, Taiwan admin-1 from a fork, thanks Chunshek!
  • Merge in modified cities in China, Taiwan from a fork, thanks Chunshek!
  • Merge in modified 10m lakes from a fork, thanks Craig!
  • Resolve ne_10m_railroads_beta1_north_america_original.shp
  • Finish adding continent codes on 10m roads, this might involve adding topology to crack geoms.
  • Include PRJ files with each GeoTIFF, for Jody and the uDig app on OSGeo disk.
  • Update quickstart MXD, QGIS docs with new file names, paths, field names / values.

Antrophogenic transformation of the terrestrial biosphere

Sunday, October 7th, 2012

[Editor’s note: The new “anthropocene” age has been detailed by the likes of The Economist, National Geographic Magazine,  The New York Times, and Wired. While much has been made of miles of road and general interconnected transportation network,  population density, and other measures,  I’m most captivated by this newish map showing “years of intensive use”. Get your Jared Diamond out and study this map. Thanks Hugo! More maps at Ecotope, thanks Andrew!]

Perhaps the most obvious mark we’ve made to the planet is in land-use changes. For millennia, humans have chopped down forests and moved rock and soil for agriculture and pastureland—and more recently, for construction.

Antrophogenic transformation of the terrestrial biosphere

CREDIT: ERLE ELLIS, ADAPTED FROM E. ELLIS, PROCEEDINGS OF THE ROYAL SOCIETY A, 369:1010 (2011) From the Science article “A global perspective on the anthropocene” DOI: 10.1126/science.334.6052.34

2012 Mountain Cartography Workshop presentations from New Zealand

Wednesday, September 12th, 2012

The 8th ICA Mountain Cartography Workshop was held in Tongariro National Park, New Zealand (map) during 1-5 September 2012. A couple dozen participants gathered to talk about maps, mountains & enjoy outdoor recreation. Martin, Tom, and I joined from the United States and other participants ranged from Argentina to Norway and the usual contingent from Switzerland and Austria. We had wonderful hosts in New Zealand organizing committee.

I’ve gathered our presentations together here as a reference, enjoy.

PDFs are linked below. Original Powerpoints over there »

Aileen Buckley: NAGI Fusion Method
Download PDF (2.6 mb) »

Benjamin Schroeter: Glacier variations – Projects at the Institute of Cartography, TU Dresden

Download PDF (22 mb) »

Martin Gamache: Cartography beyond the Planimetric
Download PDF (40 mb) »

Nathaniel V. KELSO: Cartography at Stamen Design
Download the PDF (16 mb) »

Geoff Aitken: The Tramper’s Map of the Tararua Mountain System circa 1936
Download PDF (2.7 mb) »

Dusan Petrovic: Designing photo-realistic and abstract mountain maps for a 3d mapping study
Download PDF (3.7 mb) »

Martin Gamache: Americans on Everest – 50 year anniversary, mapping with the iPad
Download PDF (28 mb) »

Roger Wheate: Visualization of changes in the alpine glaciers in western Canada
Download PDF (8 mb) »

Lorenz Hurni: Glacier DEM reconstruction based on historical maps: A semi-automated approach
Download PDF (5.6 mb) »

Sebastian Vivero: A new digital terrain model for the Tasman Glacier, New Zealand, using digital photogrammetry techniques.
Download PDF (2.7 mb) »

Karel Kriz: User interaction and design issues for the Tyrolean AWC Portal
Download PDF (1.8 mb) »

Stefan Raeber: Panoramic Maps – Evaluating the usability and effectiveness
Download PDF (2.8 mb) »

Karel Kriz: Decomposing an exhibition map
Download PDF (1.4 mb) »

Roger Smith: Texture maps with Photoshop
Download PDF (1.9 mb) »

Georg Gartner: Putting emotions in maps – Towards supporting wayfinding
Download PDF (2.7 mb) »

Karel Kriz: Needs, concepts and realization of a mountain compliant smartphone app
Download PDF (2.5 mb) »

Antoni Moore: Multi-modal exploration of rugged digital terrain on mobile devices
Download PDF (0.5 mb) »

Martin Gamache: Relief approaches at National Geographic Magazine
Download PDF (70 mb) »

Roger Smith: Maps & geomorphology
Download PDF (8.8 mb) »

Geoff Aitken: New topographic mapping
Download PDF (2 mb) »

Nathaniel Vaughn Kelso: Create your own terrain maps
Download PDF (12 mb) »

Tom Patterson: Mountains unseen – Developing a relief map of the Hawaiian seafloor
Download PDF (5.7 mb) »

Martin Gamache: Yosemite National Park and El Capitan
Download PDF (7 mb) »

Martin Gamache: Animation of projections used in the National Geographic Mapping the Oceans supplement
(view video above)

Andrew Steffert: Lidar flood mapping
Download PDF (32 mb) »

Announcing Natural Earth v2.0.0, release candidate 2

Monday, August 20th, 2012

The 2.0.0 release focuses on 6 major areas and is available to download today as preview release candidate 2.

  1. Economic geography: added global roads, railroads, ports, and airports to show how people are interconnected and routes goods move.
  2. Remastered geometries: fixed topological errors at 1:10 to 1:1,000 scales in the basic coastline, ocean, land, admin-0, and admin-1 related themes. By removing self-intersections, sliver polygons, and adjusting offset polygons, Natural Earth will import into more GIS software (like PostGIS) and be easier to maintain. The coastline was also adjusted to better conform to ~1:3,000,000 satellite imagery. Because of all these changes, some raster themes were also updated.
  3. New file name and field name schemas. Full adoption of ne_10m_theme_name.shp file names with `ne_` prefix to allow better import into GeoDB and PostGIS storage, lowercase field (column) names instead of MiXeD and UPPER cased names, and use of consistent `name` field (versus name1).
  4. Address user submitted bug reports, ~20 since the 1.4 release.
  5. Moved to Github for the backend versioned file management and coordination.
  6. Adopt semantic versioning. Know, by theme, if new changes in Natural Earth affect your project. See below for more information.

Downloads:

Please submit bugs or other oddness found release candidate 2 to myself nathaniel@kelsocartography.com by September 10th. The plan is to cut a final 2.o.0 release by the end of September. A full change log will accompany the final release.

Sample images:

Global roads:

London, Paris, Amsterdam

Brazil

New scale ranking on North American roads. Better matches with global roads, and OpenStreetMap:

DC and NYC with extra roads

Administrative units 0 and 1 have been completely remastered:

New admin-1 (states, provinces) for Natural Earth 2.0

Urban areas are now scaleranked, and false urban areas removed (mostly Nevada desert errors):

New urban scale ranks, Tokyo

Semantic Versioning

The 2.0 release in 2012 marked the project’s shift from so-called marketing versions to semantic versioning.

Natural Earth is a big project with hundreds of files that depend on each other and the total weighs in at several gigabytes. SemVer is a simple set of rules and requirements around version numbers. For our project, the data layout is the API.

  • Version format of X.Y.Z (Major.Minor.Patch).
  • Backwards incompatible changes, increment the major version X.
  • Backwards compatible additions/changes, increment the minor version Y
  • Bug fixes not affecting the file and field names, patch version Z will be incremented.

Major version increments:

  • Changing existing data file names
  • Changing existing data column (field) names
  • Removing FeatureCla field attribute values
  • Additions, deletions to admin-0
  • Introduce significant new themes

Minor version increments:

  • Any shape or attribute change in admin-0
  • Additions, deletions, and any shape or attribute changes in admin-1
  • Additions, deletions to any theme
  • Major shape or attribute changes in any theme
  • Adding, changing FeatureCla field attribute values
  • Introduce minor new themes

Patch version increments:

  • Minor shape or attribute changes in any theme
  • Bug fixes to shape, attributes in any theme

Under this scheme, version numbers and the way they change convey meaning about the underlying code and what has been modified from one version to the next.

When we introduce a new version of Natural Earth, you can tell by the version number how much effort you will need to extend to integrate the data with your map implementation.

  • Bug fixes Z: can simply use the new data files, replacing your old files.
  • Minor version Y: limited integration challenges.
  • Major version X: significatnt integration challenges, either around changed file strucutre, field layout, field values likeFeatureCla used in symbolizing data, or significant new additions or significant changes to existing themes.

A friendlier PostGIS? Top three areas for improvement

Friday, August 10th, 2012

I prompted a flurry of PostGIS hate (and some love) on Twitter last week, documented via Storify.

I’ve been using PostGIS for around 2 years now both at Stamen and before that at The Washington Post. Let me say upfront that PostGIS is amazing and is definitely in the top 5 best FOSS4G (free and open source software for geo) out there. It is a super powerful spatial data store that is free to download, install, and use (even in commercial projects!). It’s can also be mind numbingly difficult to install and use.

It doesn’t matter how awesome something is unless it’s usable. If we want the FOSS4G community to grow and be adopted by more everyday users of GIS and general users for spatial data needs, we need to improve this situation. “Patches welcome” is a programmers crutch. Actually following up with user’s real world issues is where it’s at.

Besides the specific issues outlined below, PostGIS lacks basic functions required for spatial analysis found in ArcToolbox. Those are slowly being rolled out as sidecar projects running on top of PostGIS, and CartoDB is a good case in point. But unless you’re a programmer and can roll your own (and your project budget can afford it), that’s a #fail.

@PostGIS asked me for details on how it could be friendlier and I’ve itemized around 20 below.

Top 3 areas for PostGIS improvement

1. EASIER TO INSTALL

If a project that is considered core to the FOSS4G stack (eg Mapnik, PostGIS, etc), the project needs to act like it.

Our servers at Stamen run Ubuntu Linux and we have a variety of them, running different combinations of applications and operating systems. Our staff machines are Mac laptops. There’s some pretty good installers now for Windows and Mac it seems. But the Ubuntu support has been out of sync too often.

  • Request 1a: Core FOSS4G projects should be stable and registered with official, maintained APT Ubuntu package list.

Distributing via private PPAs that are hard for end-users to discover and more cowboy in robustness is poor practice.

  • Request 1b: The APT package distribution of core FOSS4G projects should work with the last 2 versions (equivalent to 2 years) of Ubuntu LTS support releases, not just the most recent cutting edge dot release.

As of today, the latest LTS is 12.04, before that is 10.04. We just upgraded to 12.04 and are slowly upgrading the rest of our FOSS4G stack. This type of staggered versioning is standard in production environments.

While it’s nice to have cutting edge features, we also need to acknowledge that one app’s cutting edge features & cutting edge dependencies are another end-user’s dependency hell when installed with other software in the FOSS4G stack.

What’s amazing about Ubuntu is that they tell you up-front exactly how long they plan to support a particular version, in months and years (view timeline).

There should be an overlap period between the versions distributed in major package systems and the versions supported by the developers themselves, as well as an overlap with the release cycle of a system like Ubuntu. For instance, Mapnik is now thankfully in this state but for a long time supported the widely available 0.7 release inconsistently, but 0.7 was the only version widely available.

  • Request 1c: Backport key bug fixes to the prior release series.

We’ve all been burned by FOSS4G maintainers when say they’ve fixed problems in newer versions, but they don’t back port those changes to point or patch releases that are still compatible with LTS. Sometimes it’s unavoidable. Most of the time it’s not.

2. EASY DATA IMPORT, EXPORT, and BACKUP

Once PostGIS is installed it should be 100% usable without learning additional magic workflow. The existing workflow might seem normal to a Unix nerd or pro DBA administrator, but it’s not intuitive for a new user.

 2.1: IMPORT & EXPORT

I should be able to import shapefiles, the defacto geodata format, easily like this:

shp2pgsql import.shp

Instead of:

shp2pgsql -dID -s 900913 import.shp <destination_table> | psql -U <username> -d <my_new_db_name>

How to get there is detailed below. Note that the advanced power of the import flags and even the piping of raw SQL is still there if you need it as a power user. But the basic import (and export) should be that simple.

  • Request 2.1a: Include a default PostGIS spatial database as part of the basic install, called “default_postgis_db” or something similar.

This new database would be the default import location for shp2pgsql and other utilities if the user did not specify a named database. This will reduce the learning curve for new and novice users as they wouldn’t even need to create a spatial database to get up and running.

If the user needs more than one spatial database because of project managements, they can still create new spatial databases and import into those.

This would remove the requirement of becoming a postgres super user to create the first (and likely default) spatial database.

  • Request 2.1b: Include a default PostGIS Postgres user as part of the basic install, called “postgis_user” or something similar.
  • Request 2.1c: If I name a spatially enabled database in shp2pgsql that doesn’t yet exist, make one for me.

PostGIS should be making my life easier, not harder. If a database of that name doesn’t yet exist, ask if it should be made (y/n) and create it, then import. If a database is named but doesn’t have the spatial tables enabled, ask if they should be enabled (y/n) and do so.

  • Request 2.1d: It’s too hard to manually setup a spatial database, with around a printed page of instructions that vary with install. It mystifies Postgres pros as well as novices.

The support files for PostGIS’s functions and spatial reference systems have been stored in a variety of places on the file system, requiring us to remember what files to add, a search to find their location, then incantations to actually import those onto a new database to enable spatial power.

Fixed? I hear this is fixed as of Postgres 9.1 by using `CREATE EXTENSION postgis database` to create databases that are spatialized. That’s super awesome!

  • Request 2.1e: Default destination table names in shp2pgsql.

The required destination_table should be optional if I want to not use the filename of the shapefile as the table name:

shp2pgsql -dID -s 900913 import.shp <destination_table> | psql -U <username> -d <my_new_db_name>

Could be:

shp2pgsql -dID -s 900913 import.shp -U <username> -d <my_db_name>

  • Request 2.1f: Automatically pipe the output to actually put the raw SQL results into PostGIS.

The | (pipe) in the shp2pgsql command workflow is confusing. Pipe it automatically for me. I know this is a Unixism. It’s also super confusing to new users.

  • Request 2.1g: If my shapefile has a PRJ associated with it (as most do), auto populate the -s <srid> option.

It’s 2012, people. Map projections are a fact of life that computer should be making easier for us, not harder. Manually setting projections and transforms should be a last resort for troubleshooting, not every day routine.

Right now I must manually look up what EPSG code is associated with each shapefile’s PRJ file and set that using the -s <srid> flag so that SRID is carried over to the spatial database. When this is not provided, it defaults to -1.
  • Related 2.1h Projection on the fly: If you still can’t reproject data on the fly, something is wrong. If table X is in projection 1 (eg web merc) and table Y is in projection 2 (eg geographic), PostGIS ought to “just work”, without me resorting to a bunch of ST_Transform commands that include those flags. The SRID bits in those functions should be optional, not required.
  • Request 2.1i: Reasonable defaults in shp2pgsql import flags.

Your mileage may vary, but everyone I know uses the following flags to import data: -dID.

Make these the default. Add warnings and confirmation prompts as appropriate.

  • Request 2.1j: Easier creation of point features from csv or dbf.

This is a basic GIS type operation. Now I need to import manually into a table and use SQL to create the point geometries detailed here.

2.2: DATA BLESSING

If PostGIS’s claim to fame is as a spatial data store, and no more, it needs to get better at accepting all data, and releasing it to the wild again.

I often get invalid geometries reporting from PostGIS on import of geo data that works perfectly fine in Mapnik, OGR, QGIS, ArcGIS, and other GIS applications. PostGIS is too obsessive.

  • See Section 3 below for more specific requests.

I’m still researching PostGIS 2.0 to understand whether this has all been fixed. It sounds like it’s been partially fixed in that it’s now easier to “bless” the geometry into a structure PostGIS likes better, but the underlying problems seems to remain.

2.3. DATA BACKUP

  • Request 2.3a: Forward compatible pgdumps. Dumps from older PostGIS & Postgres combinations should always import into newer combinations of PostGIS and Postgres.

Data should not be trapped in PostGIS. We need an easy, transparent, forward compatible method of backing up data in one PostGIS database and restoring it into a new PostGIS database, either on the same machine, or a different machine, or the same machine with an upgraded version of PostGIS.

I should be able to backup data from PostGIS and have it be restored into newer copies of PostGIS without a problem (I constantly have this problem, especially between Postgres 8.3 and 8.4, maybe it’s fixed in Postgres 9.x and PostGIS 2.x?). I should be able to upgrade my DB and machines without it complaining.

  • Request 2.3b: Offer an option to skip PostGIS simple feature topology checks when importing a pgdump.

PostGIS might approach this with a two-pronged system. If there’s a problem with the data, it could keep around the original version untouched alongside a cleaned-up interpretation, and be able to dump either on request. Or, there could be a flag on a geometry row that specifies whether or not strict interpretation is applied. Defaulting to strict makes sense to us and maintains backwards compatibility with old version, but offers an escape hatch for data funk with topology and other PostGIS errors. This is especially troublesome for the Natural Earth data, which is slowly being edited to conform with PostGIS’s overly “right” view of the world.

3. “INVALID” GEOMETRIES, AND POINTING THE FINGER AT GEOS

Falling under the heading: “Beauty is in the eye of the beholder”: Real world data has self-intersections and other geometry burrs. Deal with it. I often get invalid geometries reporting from PostGIS on import of geo data that works perfectly fine in Mapnik, QGIS, ArcGIS, and other GIS applications. PostGIS is too obsessive.

  • Request 3a: Topology should only be enforced as an optional add on, even for simple Polygon geoms. OGC’s view of polygon topology for simple polygons is wrong (or at the very least too robust).

I understand that PostGIS 2.0 now ships with a clean geometry option. Woot woot. I think there are underlying issues, though.

  • Request 3b: Teach PostGIS the same winding rule that allows graphics software to fill complex polygons regarding self-intersections. Use that for simple point in polygon tests, etc. Only force me to clean the geometry for complicated map algebra.

ArcGIS will still let you join points against a polygon that has self intersections or other topology problems. Why can’t PostGIS?

  • Request 3c: Teach OGC a new trick about “less” simple features.

The irony of the recursive loop:

  1. PostGIS points finger at GEOS topology.
  2. GEOS (JTS) topology is based on OGC Simple Feature spec.
  3. OGC Simple Feature spec is based on an overly simplistic view of the world. It might be convenient in the expedient programing sense, but it’s not practical with real world data.
  4. Everyone at OpenGeo is happy as it’s self consistent.
  5. It’s hard for other people with real world data to actually use the software, sad face.
  • Request 3d: Beyond the simple polygon gripe, I’d love it if GEOS / PostGIS could become a little more sophisticated. Adobe Illustrator for several versions now allows users to build shapes using their ShapeBuilder tool where there are loops, gaps, overshoots, and other geometry burrs. It just works. Wouldn’t that be amazing? And it would be even better that ArcGIS.

Announcing Field Papers (Stamen)

Tuesday, May 29th, 2012

Republished from Stamen.

We’ve just rolled out a new way for you to make atlases of the world, called Field Papers. Field Papers allows you to print a multipage paper atlas of anywhere in the world and take it outside, offline, into the field. You can scribble on it, add features, or make notes about the area, all without a GPS or complicated GIS software.

Once you’ve annotated your atlas, you can upload photographs of each page back into the system to transcribe your notes into digital form. Each atlas gets its own page on Field Papers, and a simple history of edits and activity which you can share with friends or colleagues, and download for later analysis. Take a look at some of the atlases that have already been created on the Watch page, or browse by place, like France or Liberia.

The interface looks like this:

This project is a continuation of Walking Papers, which was built for the OpenStreetMap (OSM) editing community. Field Papers allows you to print multiple-page atlases using several map styles (including satellite imagery and black and white cartography to save ink) and has built in note annotation tools with GIS format downloads. You can also create a Field Papers account to collect any atlases you create or snapshots you upload, or you can stay anonymous. Maps from the two systems work together if you want OSM editing (see below).

Field Papers also offers several automation and map customization tools for more geo-inclined people, and the open source code is on Github.

Why not try making an atlas at Field Papers?

View Project »

Negative Equity Rates: How Does Your Neighborhood Fare? (Zillow)

Tuesday, May 29th, 2012

[Editor’s note: Where are home loans underwater? A thing I made at Stamen with Zillow is live, view your zipcode’s status on our interactive map! We’re using a UTFGrid to power the interactivity with new backend technology via Mapnik 2 and TileStache and frontend Leaflet.js map library to tie it all together over a Bing basemap. The assets were all statically generated and cached on S3 with a content distribution network in front of it. The piece has received coverage in the Wall Street Journal and other publications.]

Republished from Zillow.

Today Zillow made waves by becoming the most broad and accurate public source of negative equity information. We’re now able to tell, down to the ZIP code level, what the rate of negative equity is, how far underwater homeowners are, and the delinquency rate of underwater homeowners.

To make this data available to each and every one of you, we’ve worked with Stamen Design to create an interactive tool to allow you to explore the rates of negative equity in your local area. Just enter in your county, city, or ZIP code to narrow into your area. Click on the image below to enter the interactive map.

Watercolor, Terrain, and Toner tiles from Stamen

Wednesday, March 21st, 2012

Reminiscent of hand drawn maps, Stamen’s new watercolor maps apply raster effect area washes and organic edges over a paper texture to add warm pop to any map.

We’ve launched maps.stamen.com to showcase these new maps as well as our Terrain and Toner map styles.

We’d love to see these maps used around the web, so we’ve included some brief instructions to help you use them in the mapping system of your choice. These maps are available free of charge but with attribution. Details at any of the links above.