Tumblelog by Soup.io
Newer posts are loading.
You are at the newest post.
Click here to check if anything new just came in.

October 06 2010


Pushing the Limits of What a Wiki Can Do with Councilpedia

Barely two decades into the digital age, we take online media for granted. So much is so easy and convenient -- at our fingertips -- that we can forget technology can only do so much. Then we come up with a great idea that leaves us with the challenge of how to successfully push the limits.

This is what has confronted Gotham Gazette as we move into the final stages of creating our Councilpedia site. Councilpedia, a Knight News Challenge winner that I've blogged about here previously, will explore more fully the links between money and politics in New York City.

Councilpedia will enable visitors to the site to share what they know about politicians and their donors. It is to be powered by MediaWiki to let people flag something -- noting, for example, that one contributor to a candidate owns land she hopes to get rezoned for a Walmart. Gotham Gazette staff will then confirm -- or delete -- the comment.

Filtering Data

The core of Councilpedia is information already on Gotham Gazette, information from City Council (on earmarks, for example) and, above all, the massive records from the city Campaign Finance Board on giving and spending. The sheer magnitude of all this data has posed an array of problems.

The city data, while thorough and accessible, is inscrutable to most New Yorkers -- a list of largely meaningless names. To make it easier to search and understand, we set out to code the data (to indicate large donors, those from the city, unions, real estate industry etc.). With some candidates having thousands of contributors, this presented a massive task. Fortunately, we had some conscientious interns this summer who, between their other reporting responsibilities, dutifully researched and coded line after line of information under the supervision of our city government editor, Courtney Gross.

Readers will be able to examine this data in a number of ways. They can view by candidate. They can find out who else the contributor helped fund. They can look at intermediaries and determine whose money they bundled and then who it went to. And so on.

For the wiki, though, this mountain of information has been a bit much. When technical manager William JaVon Rice began uploading the data into spreadsheets he had created, the process took 36 hours and produced some 31,000 pages -- a sure indication no one would ever attempt this in print. The system balked, overwriting pages, for example, which required Rice to check every candidate's list of often hundreds of contributors to determine which ones had been overwritten. Then he had to undo the overwrite.

Pushing The Limits of MediaWiki

We're still planning to have this ready to show you in the next several weeks. And we think you'll be impressed. Not to boast, but the reporters, campaign finance aficionados and followers of city government who viewed our test felt that way.

But we do see a number of issues looming ahead. Councilpedia is intended as a living, breathing site, meaning data will continue to accumulate as officials collect more money, award more earmarks, pass more bills, and so on. The updating poses a challenge for a small non-profit like Gotham Gazette.

The magnitude of the new information -- added to the volumes we already have -- is likely to push the limits of MediaWiki even further.

With this in mind, we're looking for ways to automate the process more. And we hope someone -- any takers out there?-- will make MediaWiki more robust or create or an alternative.

As always, we appreciate your ideas, so feel free to share them in the comments below. And stay tuned for Councilpedia.

July 16 2010


Councilpedia In Private Pilot, Overcoming Tech Challenges

Over the last several months, Gotham Gazette has made major strides on its Councilpedia project, which will help New Yorkers keep tabs on their local officials and share their knowledge with others. Over the last year, the project has evolved and -- we think -- improved from our original plan.

Currently we have a pilot for the site with the design, the structure and information for three office holders. We are not ready to release this to the world, but if you would like a sneak preview please email me at grobinson at gothamgazette.com.

Councilpedia Brings City and Candidate Information to Life

Councilpedia intends to bring an array of information about City Council members and other city officials -- the bills they sponsor, background information, member items (a.k.a earmarks) -- to one site, along with campaign finance information. New York City, which has public financing of campaigns, requires a lot of disclosure on the part of candidates as to where they get their money and how they spend it, but the information can be hard to read and comprehend.

That is one way Councilpedia will be useful. First, it sorts the donors by various categories, such as unions, major givers and intermediaries. By having the campaign finance information along with voting information, Councilpedia can help people make possible connections between money and politics. They can then comment on the site.

The city information on donors is essentially a long list of names. Councilpedia will enable readers to identify who those people are. One example would be that John Doe, who gave to candidate X, owns a lot in the candidate's district and wants it re-zoned.

Anyone who registers can -- and is urged to -- comment. Gotham Gazette staff will review comments, verify them and use the leads from our readers to inform our reporting. Overall, we hope Councilpedia will enrich the debate about money and politics in New York.

Making Tools Work

In putting this project together we have grappled with adapting two disparate -- and balky -- technical tools to our needs.

The first was the
Campaign Finance Board
information. While the board provides a wealth of information (and has a very helpful staff to boot), the information can be hard to read and is not formatted the way we wanted it.

After trying various techniques to import the data, we eventually confronted the cold reality: The only feasible way -- given our limitations -- to create an attractive, user friendly site that did what we wanted it to do (and what we promised Knight it would do) was to re-input the data and code it ourselves.

This is incredibly painstaking. Luckily, we have several excellent interns this summer who pitched in to help.

The other issue dealt with adapting the wiki to our needs. While our technical manager, JaVon Rice, has pushed the limits of the wiki, we found there were things it would not do. For example, we had hoped to flag items that have recently received comment and have the comments appear along with the item.

Instead, we will have comment pages. We will indicate if a comment has been posted on a contribution or piece of legislation, but that also will not be automatic. Gotham Gazette staff will have to mark the item themselves.

Keeping the site current will also require staff intervention -- to add bills, to update financial reports, to remove offensive or simply incorrect comments.

Will it be worth it? We certainly hope so and are eager to move to the next step and engage New Yorkers in this conversation about money and politics.

Older posts are this way If this message doesn't go away, click anywhere on the page to continue loading posts.
Could not load more posts
Maybe Soup is currently being updated? I'll try again automatically in a few seconds...
Just a second, loading more posts...
You've reached the end.
Get rid of the ads (sfw)

Don't be the product, buy the product!