Posts Tagged ‘data’

« Older Entries

New Flickr shapefile public dataset 2.0 (find the esri type .shp here)

Tuesday, January 18th, 2011

2971287541_27e6a06a21

An updated version of the Flickr shapefile public dataset (2.0) was released last week. From nils official post:

… We haven’t completely forgotten about shapefiles and have finally gotten around to generating a new batch (read about Alpha Shapes to find out how it’s done). When Aaron did the first run we had somewhere around ninety million (90M) geotagged photos. Today we have over one hundred and ninety million (190M) and that number is growing rapidly. Of course lots of those will fall within the boundaries of the existing shapes and won’t give us any new information, but some of them will improve the boundaries of old shapes, and others will help create new shapes where there weren’t any before. Version 1 of the dataset had shapes for around one hundred and eighty thousand (180K) WOE IDs, and now we have shapes for roughly two hundred and seventy thousand (270K) WOE IDs. Woo. The dataset is available for download today, available for use under the Creative Commons Zero Waiver.

True to it’s claim, the version 2.0 release brings added fidelity on existing shapes (they are becoming more conformal to the features’ true geographic shape as more human sensors perambulate) and surveys some more cities and significantly more neighborhoods. From a data analytics perspective, I wish the new version had the summary photo count and centroid XY per feature of the 1.0 version. But very excited to see a new version released! Image above by Aaron Straup Cope. More coverage of things Flickr on Kelso’s Corner »

While the dataset is distributed in GeoJSON format, that isn’t accessible to everyone so I’ve mirrored an ESRI Shapefile version of the Flickr Shapefile Public Dataset 2.0 with this blog post (~60 mb). Details on how I did the conversion after the jump.

(more…)

Tags: , , , , ,
Posted in Data source, General, Geography, Mapping | 3 Comments »

The Map as Art: Contemporary Artist Explore Cartographically (Katharine Harmon)

Tuesday, October 27th, 2009

[Editor’s note: I picked up this fascinating read while in San Francisco earlier this month and devoured the artwork and critical essays by Gayle Clemans on the flight back to DC. Features pieces by Maya Lin and Paula Scher previously mentioned here. Thanks Jag!]

Artists & designers using the map medium for experimental art & innovation http://su.pr/2sijN4

Republished from BrainPickings.

What tattoo art has to do with fashion, vintage atlases and Nazi concentration camps.

We’ve always been fascinated by maps — through various elements of design, from typography to color theory to data visualization, they brilliantly condense and capture complex notions about space, scale, topography, politics and more. But where things get most interesting is that elusive intersection of the traditional and the experimental, where artists explore the map medium as a conceptual tool of abstract representation. And that’s exactly what The Map of the Art, a fantastic Morning News piece by Katharine Harmon, examines.

Corriette Schoenaerts, ‘Europe,’ 2005

Schoenaerts, a conceptual photographer living in Amsterdam, constructs countries and continents out of clothing.

Qin Ga, ‘Site 22: Mao Zedong Temple,’ 2005

In 2002, China’s Long March Project embarked upon a ‘Walking Visual Display’ along the route of the 1934-1936 historic 6000-mile Long March, and Beijing-based artist Qin kept tracked the group’s route in a tattooed map on his back. Three years later, Qin continued the trek where the original marchers had left off, accompanied by a camera crew and a tattoo artist, who continually updated the map on Qin’s back.

Continue reading at BrainPickings . . .

Tags: , , , , , , , , , , , , , , , , , , ,
Posted in art, Design, General, Geography, Mapping, Maps in the Wild, Merchandi$e, Print, Promote | Comments Off on The Map as Art: Contemporary Artist Explore Cartographically (Katharine Harmon)

The Importance of Building Geospatial Infrastructures (ESRI)

Tuesday, April 21st, 2009

[Editor’s note: Having a consistent set of quality geospatial databases enables cartographers and GIS professionals to focus on moving projects forward, avoiding costly database re-creation. Sharing the same base data across multiple projects and multiple organizations allows interoperability effeciencies and synergized results. Tom Patterson, Dick Furno, and I are working together to build a vector complement for Tom’s Natural Earth world raster landcover data at 1:15,000,000; 1:50m; and 1:110m scales. We hope these new shapefiles will play an important role in the community to create a small-scale (world, continental, and regional) geospatial infrastructure.]

Republished from ESRI’s ArcUser.
March 2009 article also available in PDF format.

Like no other time in recent history, our world is challenged. Disease, environmental deterioration, disasters, and now the widespread disruption of financial markets test the resourcefulness of society.

Over the past 40 years, GIS has evolved from a tool for managing projects to a framework for understanding and responding to problems on scales ranging from the local to the global. The geographic approach has become an important methodology for integrating data and information and enabling better decision making. The availability of quality geospatial data, together with improvements in software and hardware performance, has made these advances possible.

click to enlarge
The Solar Boston Web site uses geospatial data and high-performance, application-focused Web mapping to encourage the adoption of solar energy in the city of Boston, Massachusetts. Visitors can use tools at the site to calculate the solar potential of building rooftops and the annual cost savings that could be realized from installing solar panels.

With the move to an object-oriented platform, ArcGIS is better able to abstract and model the world, representing and integrating information about complex systems and modeling their behaviors. This is true whether the subject under study is as broad as an ocean or limited to a neighborhood.

The development of spatial data infrastructures (SDIs) represents the next logical step in the expansion of GIS use for data management and decision support. SDIs use accepted data and metadata standards in the creation of well-documented foundation datasets. Used with constantly updated operational data, SDIs make data more accessible and useful for specific tasks and analyses and save time while sharing costs. SDIs, together with GIS software, unlock the information contained in the terabytes of measurements, images, transactions, and other data stored in digital form by placing it in a geographic context.

The phenomenal growth of the Internet has multiplied the value of SDIs by enhancing the dissemination of data and information products. The newest release of ESRI software, ArcGIS 9.3.1, is focused on making information more consumable using the Internet. It supplies tools for configuring and deploying responsive and informative Web maps that help users accomplish specific tasks.

In February 2009, the Statistical Office of the European Communities (Eurostat) awarded a contract for the development of the technical components of a Web-based GIS. The contract went to a consortium that based its solution on ESRI technology.

These components will comply with the provisions of the Infrastructure for Spatial Information in Europe (INSPIRE). In establishing INSPIRE, the European Commission recognized the importance of quality georeferenced information to the understanding of the complex interactions between human activities and environmental pressures and impacts.

Two articles in the Focus section of this issue of ArcUser magazine provide additional examples of the value of building geospatial infrastructures to address complex problems and provide tangible benefits.

Maintaining water quality is essential to the health of the environment. Although water quality monitoring has been ongoing for decades, this abundance of measurement data cannot be translated into effective regulation and remediation action if it is not accessible, placed in geographic context, and amenable to analysis.

With new tools in ArcGIS, such as the regression analysis tools, vast data inventories can be placed in a geographic context and analyzed. This scatterplot matrix diagram is used for exploring data on foreclosures to discover if there is any relationship between variables preparatory to effectively modeling it.

The staff of Region 4 of the U.S. Environmental Protection Agency developed a geodatabase that manages current and historical water quality data and allows for rapid and flexible inquiry, analysis, and dissemination of this data and the information derived from it. This geodatabase, loaded into an ArcSDE server, uses feature classes, reformatted tables, and relationship classes. Information can be viewed as layer files generated from query definitions or queried by feature. This information is available from the desktop or distributed as ArcReader projects.

The benefits accrued from a geospatial infrastructure are greatly multiplied at larger scales as demonstrated by the marine SDI developed by the Portuguese Instituto Hidrografico. The SDI created by this naval organization integrates an abundance of sea monitoring data, prediction data, navigation charts, and base data using international data format standards and data models. A fully stipulated data policy and metadata for all geospatial data ensure data quality and promote data reuse. A wide range of information products generated for public, private, and military use are widely disseminated through Web portals. In addition, the SDI provides ad hoc decision support for the navy.

As these articles show, GIS professionals will play a more important role than ever in helping understand complex systems. With the development of SDIs, GIS professionals will be better able to apply GIS to transform data into knowledge. Aided by increasingly powerful tools in GIS, they can gain a better understanding of the world’s complex systems and help develop a more sustainable future.

Related ESRI article:

A Geospatial Foundation: Public, private, and military applications flow from SDI

Tags: , , , , , , , , , ,
Posted in Best practices, Data source, Geography, Interactive, Mapping, Print, Promote, Self promo | Comments Off on The Importance of Building Geospatial Infrastructures (ESRI)

ISO global road, rail shapefile (Kelso)

Monday, March 30th, 2009

I’m in search of a super generalized but comprehensive global coverage dataset or datasets that shows major highways and rail lines, even sea lanes. You can see an example of this on Plate 21 of the National Geographic 8th Edition Atlas of the World. Do you know of one? Please shoot me a note to nathaniel@kelsocartography.com or comment here if you have a tip.

Why do I want such? I am working with Tom Patterson (of Natural Earth fame) and Dick Furno (retired from The Washington Post) to release a comprehensive, attributed GIS base map dataset derived in part from the Natural Earth physical wall map at around 1:15,000,000 scale and two other consistent and self referential datasets at approx. scales of 1:50m and 1:110m. These datasets will provide coverage that perfectly registers with the modern satellite remote sensing imagery and SRTM derived topography. Yes there is 1:1m coverage around the world but it is often out of date and too detailed for doing global, continental, and regional mapping.

We hope these open source datasets will allow everyone in the cartographic community to focus on telling the best “why” and “how” visual story about their thematic data instead of spending 50 to 70% of project time looking for or creating the vector geometry that captures the basic “where” of their thematic data.

Release is expected Fall 2009 at the NACIS map conference in Sacramento. Please check back in this space for more details as they develop.

Tags: , , , , , , , , , , , , , , , , ,
Posted in Best practices, Data source, Geography, Mapping, Mountain Carto, Self promo | 3 Comments »

Interactive: Obama’s Appointments (Kelso via Wash Post)

Wednesday, March 18th, 2009

[Editor’s note: I did the Flash ActionScript 3 programming behind Head Count: Tracking Obama’s Appointments. This ambitious, collaborative database-driven project tracks the Obama administration’s senior political appointments and will be kept up-to-date with the latest happenings. A look at some of the interactive features you can find at washingtonpost.com/headcount.]

Interactive graphic and database by Sarah Cohen, Karen Yourish, Nathaniel Vaughn Kelso, Ryan O’Neil, Paul Volpe, Sarah Sampsel and Laura Stanton.

This project draws on concepts from these two blog posts in particular: It Ain’t Easy To Get A Newspaper To Provide Useful Data (TechDirt) and The New Journalism: Goosing the Gray Lady (NY Times).

Republished from The Washington Post

Heads Pop Up and Heads Roll: Let’s Keep Track.
By Al Kamen; Wednesday, March 18, 2009; Page A11

Today we launch Head Count, The Washington Post’s interactive database to help you keep a sharp eye on the people President Obama is appointing to the nearly 500 top positions in the federal government that require Senate confirmation. The new feature will not only tell you who they are but also help you count all the demographic beans — age, sex, ethnicity, education (elite schools or not), home states and so on.

At http://www.washingtonpost.com/headcount, you can search agency by agency to determine which jobs are still open, should your private-sector job be looking a little shaky these days. You can also search by individual to determine how many officials in this “change” administration are merely retreads from the Clinton days.

And Head Count will give some clues to help answer everyone’s perennial question: How did that fool get that great job? It will also tell you who paid good, hard money or bundled huge sums for Obama/Biden, who worked on the campaign, who had the coveted Harvard Law connection, hailed from Chicago or was a pal of Michelle Obama, Tom Daschle or Ted Kennedy.

The appointments that are tracked by Head Count do not include judges, ambassadors, U.S. attorneys or U.S. marshals. We’ll monitor those separately. Nor does the database include the many important officials who are not confirmed by the Senate. We’ll be tweaking the database as we go, adding new categories, such as veterans, and making other additions.

Loop Fans can help! If you’ve got information we could use or suggestions about how to improve the site, please submit comments and updates at the link provided on the Head Count Web site.

NOMINATING PARTY

The White House personnel logjam — also known as the Great Daschle Debacle — appears to have been broken. Team Obama’s nominations operation began at a record pace. But IRS problems sparked Health and Human Services nominee Tom Daschle‘s withdrawal on Feb. 3, leading to a general revetting of nominees that stalled everything.

The numbers are startling. Obama, by the end of his first week in office, had announced 47 nominees for senior-most jobs. He’d officially nominated 37 of them, according to data compiled by New York University’s Wagner School of Public Service Presidential Transition Project. (That number includes some holdovers.)

But in the month after Daschle’s withdrawal, the White House announced only 10 candidates for Senate-confirmed positions and formally nominated only six people.

In the next three weeks, however, the pace ramped up sharply, with 42 nominees named. Official nominations have been slower — only 27 during that time. But there were 15 last week, and we’re told there are plenty in the pipeline. As of yesterday, there were 39 Senate-confirmed individuals on the job. (That includes seven holdovers.)

The push now is to get as many nominees up to the Senate — and get confirmation for the three dozen or so already up there — before the Senate slithers out of town on April 3.

View the interactive at The Washington Post . . .

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,
Posted in Best practices, Charting, Data source, Design, Election, Flash, General, Interactive, Print, Self promo | 1 Comment »

Announcing IndieMapper (Axis Maps)

Tuesday, March 17th, 2009

[Editor’s note: The same folks at Axis Maps who brought us Finder! and Maker! from GeoCommons (blog post) announce IndieMapper, a new tool for cartographers by cartographers to make awesome maps. When released (summer 2009?), this online, Flash-based solution will fill a nitch between full bore GIS systems and manual compilation in Illustrator. I hope they’ll add an “embed this map” option for people who want to publish their maps straight to the web for their 2.0 release. Screenshots below.]

Republished from the Axis Maps IndieMapper blog.
Add yourself to their email list to get the 411 on the first release.

Note: David Heyman of Axis Maps was at SXSW Interactive 2009 this weekend in the NeoCartography: Mapping Design and Usability Evolved session.

Welcome to indiemapper.com! We’re very excited that you’ve taken the time to learn about our project. Put your email address in the subscription box below so we can tell you about indiemapper developments and most importantly… the launch!

A little about indiemapper:

  1. It’s big. We’re not satisfied with the current tools available for making maps. They’re too expensive and their cartographic functionality doesn’t always give us everything we need. We’re building indiemapper to replace those tools. It takes in shapefiles and spits out a vector file to Illustrator, just like GIS. It supports multiple projections, just like GIS. Labeling, map layout, data classification, just like GIS. If you need it to make a map, it’s in there.
  2. It’s focused. The problem with the existing tools for making maps is that they aren’t designed exclusively for making maps. You’re only using about 10% of the software to make your map (but paying for all 100%). We built indiemapper for only one purpose: making maps.
  3. It’s visual. If I want to reclassify my data, why do I need to go into the map properties dialog box, select the symbology tab, click on the data classification button that opens up a new window, move the tabs around in that new window, click OK in the new window, click OK in the first window I opened and then wait for my map to redraw to see if the classification looks good? With indiemapper, every update is live and every control is easily accessible. No more hunting. No more waiting.
  4. It’s online. Updates are available as soon as they’re released. No more waiting for service packs or paying for upgrades. And Mac users: Get ready… we are 100% platform independent. Use it on a Mac, use it on Windows, even try out Linux (you might like it!)

I could go on and on, and I will, right here on this blog. Check back here for a discussion on functionality, coding, design, cartography, all things indiemapper. We’ll be releasing some free tools along the way that we’ll want to tell you about too. Most importantly, we want your feedback so let us have it!

Top features:
  • Fast, visual editing.
  • Nothing more than 2 clicks away.
  • Rolling release with constant updates.
  • Choropleth mapping.
  • Dot density mapping.
  • Proportional symbol mapping.
  • Cartograms.
  • Unlimited undos.
  • Colors from ColorBrewer.
  • Type from TypeBrewer.
  • Basemaps from Natural Earth.
  • Map layout for print and screen.
  • Load data from shapefile and KML formats.
  • Export to vector SVG.
  • Export to JPG.
Screenshots:
PROJECTIONS:
  • Visual selection process
  • Re-project vector data on-the-fly
  • Filter by projections that preserve area / shape / direction
  • Learn more about projection best practices
  • Create custom standard lines / centering
DATA LAYERS:
  • Manage multiple thematic data layers
  • Create new layers on-the-fly from attributes of existing datasets
  • Control editability / visibility
  • Instantly access style / label options for each data layer
CHOROPLETH MAPS
  • Create both classed and unclassed choropleth maps
  • Select from built-in automatic classification routines or set your own breaks
  • Visually set manual class breaks using interactive live-updating histogram
  • Automatically select built-in ColorBrewer color ramps
  • Create your own custom color ramps
  • Every change is updated on the map instantly

Tags: , , , , , , , , , , , , ,
Posted in Best practices, Charting, Data source, Flash, Geography, Interactive, Mapping, Promote, Software | 1 Comment »

It Ain’t Easy To Get A Newspaper To Provide Useful Data (TechDirt)

Thursday, December 11th, 2008

[Editor’s note: Interesting take on getting old media to get data friendly and generous: seeing the “value of data in addition to straight reporting, and the concept of openness compared to being a gatekeeper.” Thanks Katharine!]

Republished from TechDirt. From the not-their-thing dept.

We’ve discussed in the past the idea that newspapers today need to get beyond reporting the news and also move towards opening up their data such that others can make that data useful. Newspapers have access to all sorts of interesting and useful data — but traditionally, they’ve hoarded it and only used it as a resource for editors and reporters in creating stories. However, by opening up that data to others, it could make those news organizations much more valuable. We’re seeing some movement in that direction, and recently noted that the NY Times had come out with an API for the campaign finance data it had. 

However, one thing that seems clear is that very few newspapers have the resources necessary to do this on a regular basis. The NY Times (and, to some extent, the Washington Post) seems to be willing to invest in this area, but for many newspapers, the entire concept seems foreign. Writing for OJR, Eric Ulken from the LA Times discusses how much effort it took to get the necessary resources just to build a homicide map to go along with a blogthat planned to chronicle every homicide in the LA area. If Ulken’s experience is any indication, it seems pretty clear that very, very few traditional news organizations are going to be able to pull this off. They’re just not set up to do such things. 

It seems increasingly clear that these types of innovations are more likely to come from newer news organizations who actually recognize the value of data in addition to straight reporting, and the concept of openness compared to being a gatekeeper.

Tags: , , , , , , , , , , , , , ,
Posted in Best practices, General, Interactive, Mapping, Promote | 1 Comment »

Our city in 3D (Google Lat-Long Blog)

Friday, July 18th, 2008

 [Editor’s note: Local interest to DC but promising in sharing of public GIS data.]

Reprinted from the Google Lat-Long Blog. Published: July 16, 2008

Posted by Barney Krucoff, GIS Manager, District of Columbia, Office of the Chief Technology Officer

The District of Columbia government has submitted more than 84,000 3D models to Google Earth via the Cities in 3D program. But why would a city, let alone one that is known as a horizontal city because of a strictly enforced height limit, be so eager to participate? Here’s a glimpse into our thinking in the District’s GIS department.

1. It is the right thing to do. Fundamentally, the District Government believes that data created with public funds should be available to the public. Making this data now available via Google Earth is an important step in making our data truly accessible to the public at large.

2. Because every neighborhood can benefit from 3D. Instead of modeling just a select few landmarks in exquisite detail, we wanted to model every building in every neighborhood. Economic development was a primary driver behind development of the dataset. The buildings provide the context in which to plan and debate proposed new developments. Despite our aforementioned reputation as a horizontal city, we are also a city of spires, penthouses and domes, as you can now see. As public sector mappers, we put the entire city on Google Earth, not just downtown, because every neighborhood needs planning and development. We hope that the private sector will follow suit and create rich 3D models of proposed developments as KML downloads in the future.

3. We get better 3D performance from the cloud and we don’t pay for it. Some GIS users in the DC government, have made excellent use of the data, but with the city’s current technology, the 3D data had to be used locally on high-end desktops. Frankly, the District did not have the technical capabilities for distributing nearly 100,000 3D building across the enterprise. With the data now hosted on Google Earth 4.3, we expect DC Government users to turn to Google Earth just like the public. And using the same tools as our citizens is another powerful way to connect with them and ensure the quality of their experience.

4. We want to communicate with our residents. It is important to us that citizens, particularly DC taxpayers, understand what we do. We posted the “coolest” data set DC GIS has, because now that we have your attention we want to show you all of the other stuff we do. As part of Mayor Adrian Fenty’s drive for transparency, the DC government now makes more than 200 geospatial data sets publically available. So admire the thousands of 3D buildings, but also extend your virtual tour. You can add these datasets as layers on Google Earth, and view things like wards, trails, parks, museums, building permits, fire hydrants, zoning and even things the city isn’t proud of, like calls for rodent abatement.

Tags: , , , , , , , ,
Posted in Best practices, Data source, Geography, Google Earth, Interactive, Mapping, Promote, Software | 1 Comment »

Data Visualisation Blogs You Might Not Know About

Wednesday, April 23rd, 2008

This list is lifted straight from Tom Carden’s blog earlier this month (visit his post). Thanks Peter!

 

Tags: , , , , , ,
Posted in Charting, Design, Promote | Comments Off on Data Visualisation Blogs You Might Not Know About

Zillow® Labs – Neighborhood Boundaries Shapefiles for Download

Tuesday, March 25th, 2008

zillow logo(Zillow.com via arcdigita.com) The Zillow data team has created a database of nearly 7,000 neighborhood boundaries in the largest cities in the U.S. And they’d like to share them with you! They’re sharing these neighborhoods under a Creative Commons license to allow people to use and contribute to their growing database.

Now comes the fine print: You are free to use the files in this database in applications as long as you attribute Zillow when you use it. You may also make your own changes to the database files and distribute them, as long as you provide them under the same kind of license and give Zillow attribution. The neighborhood shapes are available below, zipped up in the Arc Shapefile format.

The downloads are by state and available here:
http://www.zillow.com/labs/NeighborhoodBoundaries.htm

Tags: , , , , , ,
Posted in Data source, Mapping | Comments Off on Zillow® Labs – Neighborhood Boundaries Shapefiles for Download

« Older Entries