Tumblelog by Soup.io
Newer posts are loading.
You are at the newest post.
Click here to check if anything new just came in.

July 20 2011


Dotspotting + Embeds = Great Maps of Prisons, Crime, Pavement Dots

There are three basic parts to working with online representations of urban civic data in Dotspotting: collating the data, manipulating it, and then sharing and publishing it. Up until now, we've been focused on the first two, which makes sense. Obviously you need to be able to gather and work with the data before you can share it.

Today we're announcing the inclusion of the project's most requested feature: embedding the maps that people make into sites of their own.

Dotspotting makes tools to help people gather data about cities and make that information more legible. It's the first project Stamen released as part of Citytracking, a project funded by the Knight News Challenge.

Dotspotting's "embed/export" feature has been reworked to include the ability to generate HTML code that you can configure to your own specs, depending on how your site is formatted. Basic embed code is available in default mode, which will generate a map that looks pretty much the way it does on Dotspotting:

California state prisons on Dotspotting

There are a couple of different options in embed; so, for example, you can swap out the normal toner cartography for Bing's new (awesome) map tiles:

California state prisons on Dotspotting

We've been working with Mission Local, a news organization that reports on our home base of the Mission District, to find ways to take the lessons learned from the Crimespotting project and give this ability to local publications and advocates. The crime theme we've developed with them lets you generate maps that look like the one below, if you provide a "crime type" value in your data:

Crime June 21-28 updated on Dotspotting

And my favorite so far is the photo theme, which takes a "flickr:id" or "photo_url" field from your data (say, a set on flickr) and generates a visual mapping of where the photos are:

Dots on the pavement from flickr on Dotspotting

We're planning on releasing more of these as time goes by; if you've got ideas for a theme you'd like to see, please upload some data and get in touch!

March 15 2011


How Dotspotting Began with Colored Dots on Drains in San Francisco

sewer-250.jpgIn some ways you could say that the Dotspotting project started with San Francisco's sewage and drain system. A few years ago I started noticing some strange dot markings on the curbs of city sidewalks, directly above the storm drains like the one you see on the left

But on closer inspection, it turned out they weren't just single dots. They seemed to be dots that had been applied, rubbed off a bit, and reapplied. Like Roman palimpsests, the curbs above drains looked like reusable canvasses -- but for dots, instead of edicts. The image below is one close-up example.


This is the kind of thing that infrastructure dorks like me obsess over for a while... and then inevitably move on to something else. (My fascination with the dots waned as I became intrigued by another piece of urban infrastructure, the cable car, which I've been using on my daily commute for about six months -- and mapping on Dotspotting).

Spotting the Dotter

A few years went by and one day I happened to see a man riding his bicycle down Hyde Street in the Tenderloin. His route and mine coincided in just the right way for me to see him stop at a storm drain, reach down and do something with a pole, make a note on a hand-held device, and slowly bike down to the next drain. Traffic being what it was on Hyde, I was able to catch up with him just in time to see him spraying different colored dots on the curb, over dots the ones that were already there! I was pretty excited.

After a quick conversation later, I discovered that he worked at San Francisco's Mosquito Abatement Team. This is the part of the Public Utilities Commission that checks likely standing water locations in the city and, if they find mosquitoes, deals with them. Of course, they track all of their work on GPS and so forth (stay tuned for some hopeful mappings of that data).

But in order to keep things simple -- and to make it possible to see how long its been since a drain was inspected -- the Abatement Team coordinates their efforts so that they always know whether this inspection period is green, white, pink, and so forth. If a drain's most recent dot is the right color, it's been inspected. If not, it hasn't. It's a simple and elegant solution to a potentially onerous problem.

I love that the simple process of making the same mark over and over again, but making it slightly differently, results in this rich data set that can be understood and read if you've got the information and tools to interpret it. (As a side note, I also love having conversations with people who spray markings on the sidewalk. You think you're bothering them with your questions but I've found they're actually pretty pleased to talk to you.)

Visualizing Water Systems

Like a lot of other things, the water system is a gnarly beast that gets more interesting the more you poke at it. There's a ton of information available on the San Francisco utility website, including the awesome map of its wastewater system below -- it even includes the city's version of the Continental Divide:

Wastewater_System_Overview - 500.jpg

Back to Dotspotting

All of this is a long lead-in to the fact that Herb Dang, who runs the Operations department at SFWater, was a surprise visitor to the CityTracking Conference we've just finished cleaning up from at Stamen HQ. His presence sparked just the kind of conversation I had hoped would happen at the conference: developers interested in digital civic infrastructure talking directly with the people who hold the data and use it every day.

We learned a couple interesting things from Herb.

First, that there's a giant concrete wall around the city on the Bay side that channels all of the wastewater runoff down to the Southeast Treatment Plant (not to be confused with the Oceanside Treatment Plant that was almost renamed the George W. Bush Sewage Plant. ("Besides," locals joked, "if we name the local sewage plant after Bush, then what's left to name after Jesse Helms?")

Second, that any request for data about the location, diameter, and any other information about a public drain pipe in the city has to go through a technical review as well as a legal review. So, in addition to needing to verify that the information is correct, the water department also needs to verify that it's a legit request. You don't want people hoovering up information about drains that they could potentially slip bombs into, for example.

Every single building in San Francisco has their own set of records for water and drainage and sewer connections, and getting information on each one of these generates its own review processes. What that means is that if a team like Stamen wants to make a map of the water infrastructure near our office, we'll need to write a separate subpoena, for each connection. For. Each. Connection.

Herb estimated that, within a stone's throw from the studio where the conference was held, there were about 40,000 sewer connections. "40,000 subpoenas" became a catch-phrase for the rest of the day.

How We're Implementing This Info

In any event, I'm still catching up with all of the interesting discussions that were had during the conference, but that's a pretty good representative sample. It's also a nice way to segue into the design work we've been doing on Dotspotting, which I'll be demoing briefly at South by Southwest this week.

There are a number of pretty substantial improvements, but what I'm most excited about at the moment are some big changes to the interface and overall visual look of the thing. Mapping 311 requests for Sewer Issues in District 10 used to look like the cropped image below.

dotspotting_old - 500.jpg

Now, with some swanktastic custom cartography that Geraldine and Aaron have been working on, and visual and interactive improvements that Sean and Shawn (I know, weird, right?) have been polishing and making right, it looks more like the shot below. We'll be pushing this work live some time before my SXSW presentation.

dotspotting_new - 500.jpg

So, onwards we go. Upload yer dots!

January 20 2011


Dotspotting Expands to Track Homicides, Food Vendors, Road Trips

Since my last post, we've been busily working on extending the functionality of Dotspotting, the first project in our larger Citytracking project aimed at helping people tell stories about cities. It's still, as my colleague Aaron puts it, very much in a "super alpha-beta-disco-ball" state -- which mainly just means we don't want anyone to put data in there that they expect to keep -- but it's getting there.

A few things have happened that I want to update you about:

  • Import has been expanded from only accepting .csv files to include .gpx, .json, .kml, .rss and .xls files. Various people around the studio use a variety of GPS tracking software. I use Trails for the iPhone, Julie uses Mytracks for Android, and so on. We've been starting with those formats as a baseline, using the files that different applications export and pulling them into Dotspotting.
  • Export has been expanded to include all of the above file types, and also includes .png files. We're hoping this is going to be particularly useful for journalists who want to include images of geographic content in their articles but don't want to use the screen-capping-a-google-map-and-hoping-the-legal-department-doesn't-catch-on technique. So these kinds of images become easy to export out of the system.
  • Search is coming along. This report from DataSF about 311 activity in District 6, where the studio is, has 392 dots, showing the wide variety of calls for service that the system handles in a week. You can now search these kinds of reports fairly comprehensively, so it's now possible to make maps of only those requests having to do with Catch Basin Maintenance, graffiti, or tree maintenance. These are the kinds of queries that we want to enable journalists and others to make when telling stories about city data, and they're the kind of thing that lots of current city data services don't report, so it's gratifying to see those come together.
  • Search is also working in a limited fashion relative to position, and PDF export is next on the list. More on these next time.
  • We've squashed a lot of bugs related to importing and exporting, and found a bunch more of course. We're keeping track of these on the project GitHub account; if you find one, please let us know.
  • Uploads are starting to trickle in from outside the studio walls: Homicides in Richmond, New Food Vendors in Vancouver, and trips along the coast of California are a few of what we've seen. We're in conversations with a couple of cities and other groups, more on that next time.


December 02 2010


Dotspotting Launches to Make City Data Mappable


Dotspotting is the first project Stamen is releasing as part of Citytracking, a project funded by the Knight News Challenge. We're making tools to help people gather data about cities and make that data more legible. Our hope is to do this in a way that's simple enough for regular people to get involved, but robust enough for real research to happen along the way.

There's currently a whole chain of elements involved in building digital civic infrastructure for the public, and these are represented by various Stamen projects and those of others. At the moment, the current hodgepodge of bits -- including APIs and official sources, scraped websites, sometimes-reusable data formats and datasets, visualizations, embeddable widgets etc. -- is fractured, overly technical and obscure, and requires considerable expertise to harness. That is, unless you're willing to use generic tools like Google Maps. We want to change this. Visualizing city data shouldn't be this hard, or this generic.

So the first part of this project is to start from scratch, in a "clean room" environment. We've started from a baseline that's really straightforward, tackling the simplest part: Getting dots on maps, without legacy code or any baggage. Just that, to start. Dots on maps.

More Than Dots

But dots on maps implies a few other things: Getting the locations, putting them on there, working with them, and -- crucially -- getting them out in a format that people can work with.

We've had several interactions with different city agencies so far, and while the situation has changed alot in the last few years, we've realized that, for the foreseeable future, people aren't going to stop using Word and Excel and Pages and Numbers to work with their data, or even stop using paper. It's made us think that if this stuff is really going to work out in the long run, we need to focus our thinking on projects that can consume as well as export things that cities and people actually use and use now. This is instead of going with projects that have to rely on fancy APIs or the latest database flavor.

It's great that San Francisco and New York are releasing structured XML data, but Oakland is still uploading Excel spreadsheets (it's actually awesome that they do), and the Tenderloin police lieutenants are printing out paper maps and hand-placing colored stickers on them. At some point, if this really is the way things are going, we're going to need to meet the needs of actual functioning city agencies; and while APIs are great and necessary, for now that means Excel spreadsheets and Word docs. It also means being able to easily read in data that people have uploaded to Google maps, interface with SMS systems like those that Ushahidi are pioneering. And it means being able to export to things like PowerPoint and Keynote, scary as that may seem.

What we've launched with is the baseline work that's being done to make this stuff internet-native. There's a login and permissions system that pretty much works. Uploading .csv files full of dots works. Each dot has an HTML page of its own, for example, like they do on Crimespotting. Collections of dots (we're calling them sheets) work, and you can export them. And there are dots on maps.

Easter Egg

What's up with the funny map, above, you ask? That's an undocumented Easter egg that allows you to change the default base map for Dotspotting on the fly using a templated URL. If that last sentence sounds like gibberish, just think: Fun! And a bit dorky. But fun!

Speaking of which, the code for Dotspotting is available for download on Github, and licensed for use under the GNU General Public License. We're planning on releasing the code as we work on the project, in the hope that working in this kind of transparent manner from the beginning will both benefit the project and serve as an example of the way we'd like to work with foundations on this kind of work.

Older posts are this way If this message doesn't go away, click anywhere on the page to continue loading posts.
Could not load more posts
Maybe Soup is currently being updated? I'll try again automatically in a few seconds...
Just a second, loading more posts...
You've reached the end.

Don't be the product, buy the product!