Archive for the ‘Data source’ Category

Haiti OpenStreetMaps + Google Map Maker

Thursday, January 21st, 2010

There are two camps opening up in the Volunteered Geographic Information (VGI or community generated content CGC) community that are sure to have big import going down the line. One is OpenStreetMap.org (editor | download) and the second is Google’s Map Maker (not available in the US and other developed world locations) (editor | tile view options | download). Both services allow users to either upload their own GPS track or digitize linework and points off satellite imagery. These two options have added to the global map, often creating the first detailed map of a place ever available to the general public.

But the two projects have very different license structures (OSM almost unrestricted, Google very restricted). Not only do they duplicate effort, but they result in “similar but different” products that do not perfectly register with each other. This is an problem that faced many in the US in the 1990s as many organizations developed their own, not-interoperable datasets for the same regions. That model has largely been replaced by single entities building fundamental datasets that other organizations and individuals repurpose.

The licensing and data coverage & registration issues are of note to professional cartographers (and first responders) more so then to the general public. But, as Sean says:

OSM and MapMaker aren’t talking and I think it is a big problem – if you want to help rescue efforts in Haiti where do you go to digitize? OSM? MapMaker?

Muki Haklay has a good comparison of the detail in each of OSM and Map Maker for Haiti. In the map here, yellow means that there is a better coverage in Map Maker, and blue means that there is a better coverage in OpenStreetMap.
Screenshot below, click for larger version.

osm-mapmaker-haiti-180110

“The comparison looks at total roads length for both datasets. The calculated difference between them using the equation:

∑(OSM roads length)-∑(Map Maker roads length)

for each 1km grid square.

The information in the file can be used for the following applications:

  • Users of these mapping products - it can help in judging which dataset to use for each area.
  • Users – it can facilitate conflation -  the process of merging datasets to create a better quality output.
  • Mappers - it can illuminate which areas to focus on, to improve coverage.”

Haitian Earthquake Emphasizes Danger of a Split Geo Community (seen over at FortiusOne’s Off the Map blog advocating for Creative Commons 0 “zero” licensing of geodata during disasters) has a overlay of OSM (Open Street Map) and Google Map Maker data.

4276602759_bbf15d1d1f

And finally, before and after images of OpenStreetMap.org map for Port-au-Prince, Haiti:

OSM just after the Earthquake

before4274264767_c9933d12c5

OSM Today

after4274264771_6873e16fa0

Adding new rivers and lakes to 10m Natural Earth in North America

Tuesday, January 19th, 2010

Tom and I have been busy adding 4 times the rivers and 3 times the lakes we had for North America. This adds in many “missing” hydro features that one might normally find on a 1:10,000,000 hydrologic reference map.

Why were they missing from the first version of Natural Earth? It’s hard to wade thru 1:1,000,000 features to figure which to add and an even tougher job to attribute them with the correct name and scale ranks. There’s another factor: these extra features are great if you’re making a watershed map, but can be a little noisy when used as a background layer in say a political reference map.

Cody Rice, now of the EPA but formerly of the Commission for Environmental Cooperation (CEC) send along an amazing link last week. The CEC is a joint agency between the United States, Canada, and Mexico. Specifically: USGS, Natural Resources Canada, INEGI-Mexico. Each country contributed base data for a 1:10,000,000 digital atlas. The data is available in many popular formats and is in the public domain. Better yet, it includes GIS data attributes like river name!

We’ve compared with our existing Natural Earth linework and identified which features were missing. For those we’re adding, we’ve adjusted the new linework a nudge here and there so it lines up with SRTM relief and our existing linework. We’ve also gone thru and created lake centerlines and applied scale ranks to all in three new steps (10, 11, and 12). We have some final polishing but will be releasing, along with some slight adjustments to the original data, by the end of January.

Do you have time to donate? Unlike ranks 0 to 9 (the original data), this new data will NOT come tapered. We’d like it to be and can show you how.

Know of a similar, attributed with name, 1:10,000,000 regional dataset we could adapt into Natural Earth to build out our coverage? Please let me know at nathaniel@kelsocartography.com.

Preview images below:

Red = new at rank 10. Blue = new at rank 11. Black = new at rank 12. Grey = old at ranks 0 to 9.

Click images to view larger sizes.

hydro_west_coast

hydro_mid_west

hydro_east

hydro_mexico

hydro_alaska

hydro_yukon

hydro_quebec

Updating Natural Earth Populated Places (Kelso)

Tuesday, January 12th, 2010

Dick Furno and I have been busy revising and expanding the populated places included with Natural Earth, released in December. We’re adding around 1,000 places to get to ~7,300 total towns and settlements around the world. We include more admin-1 capitals and make a first stab at linking to GeoNames via their unique IDs. Dick has gone over the towns and created a new 30m scale rank that helps cut the 20m clutter in half. We hope to have the update pushed out by the end of January.

Here’s a preview: Click images to see larger version

Added locations
(red = added general place, black = added admin-1 place at less than 10m scale rank, grey = existing places)

added_place_or_admin_1_capital

Scale rank changed
(red = change, grey = no change to existing or added at less than 10m scale rank)

scale_ranks

Changed the name, location, feature class, population, or other property
(blue = changes to existing features, grey = existing or added)

names_location_featureclass_population

Places with GeoName ID linkages
(green = with ID, grey = lacks ID for mostly small population settlements)

geonames_id

Natural Earth Browser from Thematic Mapping

Monday, January 4th, 2010

[Editor's note: Bjorn over at his Thematic Mapping Blog has done up a Natural Earth tile set using open source tools. How have you been using Natural Earth?]

Excerpted from Thematic Mapping Blog.

My holiday project, apart from skiing, was to play with the new Natural Earth dataset. By combing raster and vector data you can make a variety of visually pleasing maps. You can use my Natural Earth Browser to study the great linework of Natural Earth.
Natural Earth Browser was created with a variety of open source tools. Map tiles from raster data was created with MapTiler and optimised with pngng. Map tiles from vector data was styled with Mapnik and pre-genereated with TileCache. The map interface is based on OpenLayers, Ext JS and GeoExt.
Continue reading at Thematic Mapping Blog . . .

What is the difference between a sea and a lake? (Environment Canada)

Friday, December 11th, 2009

[Editor's note: This Q&A from Environment Canada explains the rough difference between types of hydrological features. Names in the real world are often messier than this text book explanation. One way we've tried to help in Natural Earth is by indicating if a lake is freshwater, saline, natural, artificial, stable water level, seasonal water level, or simply ephemeral.]

Republished from Environment Canada. Feb. 2002.

What is the difference between a sea and a lake? Looking at the names of many sea and many lakes does not readily demonstrate an identifiable difference. There are salt water lakes and fresh water seas and some lakes that are bigger than other seas.
Bruce Schoenegge, Irvine, California, USA

Salt crust resulting from receding lake, Lake Frome, Australia.
Salt crust resulting from receding lake, Lake Frome, Australia.

In order to understand why some smaller salt water bodies are called lakes and others seas it is necessary to realize that lakes are, in geological time scales, transitory in nature–they form, mature and die.

Some water bodies that started out as saltwater seas over time became closed-off from the oceans. Depending on the quantity of fresh water flowing in from rivers, glacial melt water, or other sources, the salinity could have declined to the point where the water became relatively fresh. The Sea of Aral is probably an example of where this transition has occurred. Similarly the reverse can occur whereby freshwater lakes can become open to the sea so that the salinity increases, as in the Baltic Sea. The Black Sea is an example that has alternated between fresh and salt water conditions over geological time. Evidence for these changes can be found in ancient fossils of organisms some of which were known to be tolerant of saltwater while others were known to have been intolerant.

No doubt there was also some confusion in the naming of water bodies by the early explorers based on their first impressions and certainly one can understand why some may have been inappropriately named. In addition the subsequent translation of the names between different languages could also have added to the confusion.

Here are some definitions of water bodies:

Ocean
The whole body of salt water that covers nearly 3/4 of the surface area of the globe. In particular, each of the main areas into which the sea is divided geographically, e.g. Atlantic, Pacific. Oceans are tidal, living systems containing a multitude of biological organisms.

  • Average depth of the world’s oceans: 3,962 metres(13,000ft)
  • Maximum depth: 10,680 meters; (35,040ft)
  • Average salt content – 3.5% (mostly common salt, NaCl but with some magnesium and calcium salts)
  • Average density – 1.026
Sea
The expanse of salt water that covers most of the earth’s surface and surrounds the land masses. A body of salt water that is secondary in size to oceans.
Lake
A large area of water surrounded by land. Normally fresh water but in some cases can be appreciably saline depending on the geology of the underlying and surrounding terrain. Lakes are living systems containing various quantities of biological organisms. Lakes can be classified according to the level of bioproductivity as oligotrophic (low productivity), mesotrophic or eutrophic (high productivity). The productivity is usually controlled by the amount of nutrients (mostly phosphorus and nitrogen) present in the water and the amount of light that can penetrate the water column.
River
A large natural stream of water flowing in a channel to the sea, a lake or another such river. The flow can be permanent of seasonal.
Stream
A small, narrow river flowing on the surface of, or beneath, the ground.

Natural Earth Released, Let the Downloads Begin!

Thursday, December 3rd, 2009

Tom and I are pleased to announce the immediate availability of Natural Earth, free vector and raster map data at 1:10m, 1:50m, and 1:110m scales. This is a NACIS and MapGiving co-branded product with assistance from the University of Wisconsin-Madison cartography lab, Florida State University, and others.

Do you have a new theme to contribute to Natural Earth? Great! Please follow these data creation guidelines so it fits in with the rest of the project. Find an error? Log it via the Corrections system.

Why Create Natural Earth?

We have two goals:

First, to give cartographers an off-the shelf solution for creating small-scale world, regional, and country maps. To this end, Natural Earth Vector includes both cultural and physical features and builds on Tom Patterson’s Natural Earth raster data, first introduced in 2005.

Second, we include many features missing from people’s mental map of the world in the hope of improving overall geographic literacy.

Natural Earth Vector solves a problem that many NACIS members face: finding vector data for making publishable-quality small-scale maps. In a time when the web is awash in interactive maps and free, downloadable vector data, such as Digital Chart of the World and VMAP, mapmakers are forced to spend time sifting through a confusing tangle of poorly attributed data. Many cartographers working under tight project deadlines must use manually digitalized bases instead.

Small-scale map datasets of the world do exist, but they have their problems.

For example, most are crudely generalized—Chile’s fjords are a noisy mess, the Svalbard archipelago is a coalesced blob, and Hawaii has disappeared into the Pacific two million years ahead of schedule. They contain few data layers, usually only a coast and country polygons, which may not be in register with each other or modern satellite imagery. The lack of good small-scale map data is not surprising. Large mapping organizations that release public domain data, such as the US Geological Survey, are not mandated to create small-scale map data for a small user community that includes mapmaking shops, publishers, web mappers, academics, and students—in other words, typical NACIS members. Natural Earth Vector fills this oft-overlooked but important niche.

Collaboration

Making Natural Earth Vector is a collaboration involving many volunteer NACIS members. Nathaniel Vaughn Kelso and Tom Patterson began working on the project in late 2008. Following the path of least resistance, the idea was to repurpose existing data that we already had as an integrated world dataset at three map scales.

The 1:50 million and 1:110 million-scale data comes from bases developed by Dick Furno and additional staff at the Washington Post for quick turnaround newspaper mapping— the Washington Post Legal Department kindly granted us permission to use these data. The kernel for the 1:10 million data was a compilation by Patterson for the Physical Map of the World, consisting of coastlines, rivers, lakes, and physical feature labels. Expanding and improving on this foundation has been our chief activity.

The core team grew to include Tanya Buckingham, who coordinates data attributing by Ben Coakley, Kevin McGrath and Sarah Bennett at the University of Wisconsin Cartography Lab; Dick Furno as populated places guru; Nick Springer as the website developer; and Lou Cross as NACIS liaison.

A cast of consultants, many regulars on the Cartotalk.com discussion forum, assisted with place names for various world regions. They include Leo Dillon, Hans van der Maarel, Will Pringle, Craig Molyneaux, Melissa Katz-Moye, Laura McCormick, Scott Zillmer and fellow staff at XNR Mapping.

Data for cartography

We developed a world base map data suitable for making a variety of visually pleasing, well-crafted maps. Unlike other map data intended for scientific analysis or military mapping, Natural Earth Vector is designed to meet the needs of mainstream production cartographers. Maximum flexibility was a goal. For example, Natural Earth Vector comes in ESRI shapefile format, the Geographic projection, and WGS datum, which are de facto standards for vector geodata.

Neatness counts with Natural Earth Vector. The carefully generalized linework maintains consistent, recognizable geographic shapes at 1:10m, 1:50m, and 1:110m scales. As Natural Earth Vector was built from the ground up, you will find that all data layers align precisely with one another. For example, where rivers and country borders are one and the same, the lines are coincident.

Natural Earth Vector, however, is more than just a collection of pretty lines. What lies beneath the surface, the data attributes, is equally important for mapmaking. Most data contain embedded feature names, which are ranked by relative importance. Up to eight rankings per data theme allow easy custom map “mashups” to emphasize your map’s subject while de-emphasizing reference features.

Other attributes facilitate faster map production. For example, width attributes assigned to rivers allow you to create tapered drainages with ease. Assigning different colors to contiguous country polygons is another task made easier thanks to data attribution.

Other key features:

  • Vector feature include name attributes and scale ranks – know the Rocky Mountains are larger than the Ozarks.
  • Large polygons, such as bathymetric layers, are split for more efficient data handling.
  • Projection friendly—vectors precisely match at 180 degrees longitude. Lines contain enough data points for smooth bending in conic projections, but not so many that processing speed suffers.
  • Raster data includes grayscale-shaded relief and cross-blended hypsometric tints derived from the latest NASA SRTM Plus elevation data and tailored to register with Natural Earth Vector.
  • Optimized for use in web mapping applications, such as Google, Yahoo, and OpenStreetMaps with built-in scale attributes to direct features to be shown at different tile zoom levels.

Data development

Since Natural Earth Vector is for visual mapmaking, we prepared the base layers in Adobe Illustrator in conjunction MAPublisher import and export filters. Illustrator offered us flexible tools for editing lines and polygons, organizing data on layers, and the ability to inspect the final data in a map-like form. A variety of third-party plug-in filters and scripts, some written by Kelso, were essential for linework generalization and other tasks.

World Data Bank 2 was the primary vector data source that required significant modifications. For example, we found that the entire west coast of the United States was about seven miles west of its true position and adjusted it accordingly. Slight adjustments to river positions better matched them to shaded relief derived from more satellite data. For Antarctica, we completely abandoned World Data Bank 2. Here, the coast, glaciers, and ice shelves derive from 2003-2004 NASA Mosaic of Antarctica, a MODIS product. We also updated the data to reflect recent ice shelf collapses.

Contributors from around the globe researched additional feature names beyond those original to Patterson’s Physical Map of the World. Attributing the data was performed in ArcGIS by the team at the University of Wisconsin and by Nathaniel Vaughn Kelso at The Washington Post.

Future activity

We regard the initial release of Natural Earth Vector as a starter dataset that will see periodic updates. With any project as complex as this, flaws and omissions are bound to emerge, requiring our attention. One proposal is to form a Natural Earth map data committee that will incorporate information from users, perhaps using a Wiki model, for coordinating updates. Rivers, lakes, cities, and first order admin are components still in need of refinement. Possible data for future updates include transportation (roads and railroads), time zones, and terrestrial hypsography.

If you have ideas for Natural Earth or
want to show off how you’re using the data,
please drop us a line.

Nathaniel Vaughn Kelso
nathaniel@kelsocartography.com

Tom Patterson
mtmaps@verizon.net

Online Maps: Everyman Offers New Directions (NY Times)

Tuesday, November 17th, 2009

zooatlantabeforeatlantazooopenstreetmap[Editor's note: As my music prof was want to remind, the only difference between amateur and professional is one gets paid and the other doesn't. My hope is Google Maps starts offering user-generated geodata back to the community, like OpenStreetMap.org now does. Left image is before community edits, right is after. Thanks Nora!]

Republished from the New York Times.

SAN FRANCISCO — They don’t know it, but people who use Google’s online maps may be getting directions from Richard Hintz.

Mr. Hintz, a 62-year-old engineer who lives in Berkeley, Calif., has tweaked the locations of more than 200 business listings and points of interest in cities across the state, sliding an on-screen place marker down the block here, moving another one across the street there. Farther afield, he has mapped parts of Cambodia and Laos, where he likes to go on motorcycle trips.

Mr. Hintz said these acts of geo-volunteerism were motivated in part by self-interest: he wants to know where he’s going. But “it has this added attraction that it helps others,” he said.

Mr. Hintz is a foot soldier in an army of volunteer cartographers who are logging every detail of neighborhoods near and far into online atlases. From Petaluma to Peshawar, these amateurs are arming themselves with GPS devices and easy-to-use software to create digital maps where none were available before, or fixing mistakes and adding information to existing ones.

Like contributors to Wikipedia before them, they are democratizing a field that used to be the exclusive domain of professionals and specialists. And the information they gather is becoming increasingly valuable commercially.

Google, for example, sees maps playing a growing strategic role in its business, especially as people use cellphones to find places to visit, shop and eat. It needs reliable data about the locations of businesses and other destinations.

“The way you get that data is having users precisely locate things,” said John Hanke, a vice president of product management who oversees Google’s mapping efforts.

People have been contributing information to digital maps for some time, building displays of crime statistics or apartment rentals. Now they are creating and editing the underlying maps of streets, highways, rivers and coastlines.

“It is a huge shift,” said Michael F. Goodchild, a professor of geography at the University of California, Santa Barbara. “This is putting mapping where it should be, which is the hands of local people who know an area well.”

That is changing the dynamics of an industry that has been dominated by a handful of digital mapping companies like Tele Atlas and Navteq.

Google is increasingly bypassing those traditional map providers. It has relied on volunteers to create digital maps of 140 countries, including India, Pakistan and the Philippines, that are more complete than many maps created professionally.

Last month Google dropped Tele Atlas data from its United States maps, choosing to rely instead on government data and other sources, including updates from users.

“They have coverage in areas that the big mapping guys don’t have,” said Mike Dobson, a mapping industry consultant who once worked at Rand McNally. “It has the opportunity to cause a lot of disruption in these industries.”

Continue reading at New York Times . . .

Natural Earth Vector Preview: Cities (Part 2)

Thursday, November 12th, 2009

Announced at NACIS in Sacramento, California in October, we’re closing in on final release of Natural Earth vector and raster map data.

Bill Buckingham wrapped up processing the Natural Earth Vector cities (populated places point locations) this week. I’ve been honing our admin-1 and admin-1 rankings and feature names (only 4,000 states and provinces around the world, wew!).

Bill’s added population estimates for each city based on LandScan. The technique allows the user to know both the relative “regional” importance of a town, regardless of it’s population, based on which map scales the feature should be visible (thanks to Dick Furno) at AND to know how many people live there.

By taking a composite of both, you can still show small population cities that are regionally important at a small type point size along with larger populated places at the smaller map scales.

We have about 6,500 cities in Natural Earth Vector. Over 90% of those have population estimates (the ones that don’t are usually out in the boondocks). Together, our cities capture over 3 billion people or half of humanity.

For comparison, most other populated place GIS files have only 2,000 some cities and they focus on country and first order administrative capitals with a bare smattering of other towns. For instance: Lagos, Nigeria or San Francisco, California.  This makes smaller countries with lots of administrative divisions (like Slovenia, Vietnam, or Jamaica) seems way more populated than larger countries with larger administrative divisions (like the United States). See the North America screenshot below for an example and look at the Caribbean versus United States.

They also don’t estimate populations, and if they do they use official census number that hide the true “metro”-style counting of people that should inform a thematic map regardless of formal administrative boundaries at the smaller map scales that Natural Earth excels at.

Now for some screenshots:

(Scale ranks, followed by population view color coded like the scale ranks with nodata green dots, then cyan dot version is ESRI cities overlayed)

0world_ranks

0world_population

1no_amer_ranks

1no_amer_population

1no_amer_esri

2us_ranks

2us_population

2us_esri

More continents o’ dots after the jump.

(more…)

Natural Earth Vector Preview: Cities

Tuesday, August 18th, 2009

We’re closing in on having the cities for Natural Earth Vector complete. The final compilation has been made (focusing on a universal coverage based on regional importance, even if the town has less than typical population). Dick Furno has headed up this data theme and is half way thru applying 8 scale ranks to the cities. Population estimates will be added in a final step by another contributor. Screenshots show quick plots of the GIS data. Color implies ranking.

California (below):

california

Alaska-Yukon (below):

alaska-yukon

Europe Biggest Cities (below):

europeless

Europe All Cities:

europemore

Beijing-Tokyo biggest cities (below):

beijing-tokyo

Sudan’s Border Dispute Over Abyei. Do They Agree? Yes, No, and Sort Of (Economist)

Tuesday, August 11th, 2009

cma980[Editor's note: Update your 1st order admin maps! The long simmering feud between northern and southern Sudan was partially arbitrated by The Hague last month. The south will decide in a 2011 referendum the question of seceding from Sudan. Many observers see secession getting the thumbs up. The freshly shaped Abyei state will vote on joining or staying individually while all others in the south proper will vote en mass.]

Republished from The Economist.
Jul 30th 2009 | JUBA AND KHARTOUM

A ruling from The Hague pleases the north and vexes the south

OFFICIALS in Sudan’s government in Khartoum could hardly believe their luck when, on July 22nd, the Permanent Court of Arbitration in The Hague ruled on the fate of the disputed Abyei state, which sits astride the oil-soaked border between Arab northern Sudan and the ethnically African (and largely Christian) south Sudan. Surprisingly, the court reversed an earlier commission’s ruling and redrew Abyei’s borders, snipping out the lucrative Heglig and Bamboo oilfields (see map) and giving them to the north.

The ownership of these oilfields has soured relations between the north and south Sudanese ever since a peace accord was signed between them in 2005, ending a civil war that had raged on and off for nearly half a century at a cost of some 2m lives. So tense had the situation in Abyei become that last year much of its capital was burned to the ground in fighting between militias from the two sides. Now, however, the north seems to have got what it wanted by law rather than by force.

Continue reading at The Economist . . .