Tumblelog by Soup.io
Newer posts are loading.
You are at the newest post.
Click here to check if anything new just came in.

July 01 2011

15:00

ProPublica’s newest news app uses education data to get more social

Yesterday, the U.S. Department of Education’s Office of Civil Rights released a data set — the most comprehensive to date — documenting student access to advanced classes and special programs in U.S. public high schools. Shorthanded as the Civil Rights survey, the information tracks the access schools provide to their students for offerings, like Advanced Placement courses, gifted-and-talented programs, and higher-level math and science classes, that studies suggest are important factors for educational attainment — and for success later in life.

ProPublica reporters used the Ed data to produce a story package, “The Opportunity Gap,” that analyzes the OCR info and other federal education data; their analysis found that, overall and unsurprisingly, high-poverty schools are less likely than their wealthier counterparts to have students enrolled in those beneficial programs. The achievement gap, the data suggest, isn’t just about students’ educational attainment; it’s also about the educational opportunities provided to them in the first place. And it’s individual states that are making the policy decisions that affect the quality of those opportunities. ProPublica’s analysis, says senior editor Eric Umansky, is aimed at answering one key question: “Are states giving their kids a fair shake?”

The fact that the OCR data set is relatively comprehensive — reporting on districts with more than 3,000 students, it covers 85,000 schools, and around 75 percent of all public high schoolers in the U.S. — means that the OCR data set is also enormous. And while ProPublica’s text-based takes on the info have done precisely the thing you’d want them to do — find surprises, find trends, make it meaningful, make it human — the outfit’s reporters wanted to go beyond the database-to-narrative formula with the OCR trove. Their solution: a news app that encourages, even more than your typical app, public participation. And that looks to Facebook for social integration.

The app focuses on measuring equal access on a broad scale: It tracks not only the educational opportunities provided by each school, but also the percentage of teachers with two years’ experience or less — who, as a group, tend to effect smaller attainment gains than their more experienced counterparts — and the percentage of students who receive free or reduced-price school lunch, an indicator of poverty. (More on the developers’ methodology here.)

ProPublica leads the field in developing news apps; each one requires careful thought about how users will actually navigate — and benefit from — the app. With this one, though, “we were focusing a lot more on what behaviors we wanted to encourage,” says Scott Klein, ProPublica’s editor of news applications. ProPublica thinks about how organize reporters, both within and outside of its newsroom, around its stories, notes Amanda Michel, ProPublica’s director of distributed reporting. “Here, we wanted to take it one step further.”

With that in mind, the app invites both macro and micro analysis, with an implicit focus on personal relevance: You can parse the data by state, or you can drill down to individual schools and districts — the high school you went to, or the one that’s in your neighborhood. And then, even more intriguingly, you can compare schools according to geographical proximity and/or the relative wealth and poverty of their student bodies. (Cambridge Rindge and Latin School, just down the street from the Lab, has 1,585 students, 38 percent of whom receive a free or reduced-price lunch; Medfield Senior High, a few miles southwest of Cambridge, has 920 students and a 1-percent free/reduced lunch rate. Four percent of Rindge and Latin’s students are enrolled in advanced math courses; for Medfield High, the rate is 42 percent.) “It really is an auto-story generator,” Umansky says.

And sharing — the Facebook aspect of the app — is a big part of the behavior ProPublica’s news apps team wanted to encourage. They considered social integration from a structural perspective, notes Al Shaw, the developer who authored the app, and worked with Vadim Lavrusik, Facebook’s Journalist Program Manager, to optimize the app-to-Facebook interface. One small-but-key feature: With that integration, users who are signed into Facebook can generate an individual URL for each cluster of data they dig up — the Rindge and Latin-versus-Medfield comparison, say — to make sharing and referencing the data almost seamless. The resulting page has a “share on Facebook” button along with a note: “Use this hashtag to share your insights on Twitter: #myschoolyourschool.”

The ed-data app isn’t social for its own sake; instead, it serves the broad and sometimes nebulous goal of having impact — on both a personal and a policy level. “We invest so much time into acquiring data and cleaning data and making sense of data,” Michel notes; ultimately, though, data doesn’t mean much unless people can understand how it immediately affects them, their communities, their kids. Its newest app, Michel says, is part of ProPublica’s broader strategy: to make data, overall, more social. (They’d like to do a similar integration with Twitter, too, she says.) The point is to find ways to marry social and story, to turn online interactions into their own kind of data sets so, she says, “people can layer their stories on top of them.”

November 10 2010

18:30

Talking Points Memo’s first developer talks startup life, jumping to ProPublica and data journalism

What’s it like being the only in-house techie at a news startup? Talking Points Memo’s first developer Al Shaw says “it’s kind of like being a reporter….you have to be a generalist,” doing everything from ad-side work to election-night interactives.

Shaw was the primary technical force behind most of the bells and whistles that cropped up at TPM over the past two years, including a redesign that lets producers switch up the layout of the homepage, and an array of slick interactives like the real-time election results tracker that made TPM look a lot less like a scrappy startup and more like an establishment outlet on Election Night earlier this month. (Shaw is quick to explain he had some help on the election map from Erik Hinton, TPM’s technical fellow.) He’s also been good about blogging about his technical endeavors in ways that could be useful to his peers at other news organizations.

Shaw announced last month he is leaving TPM to start a new gig at ProPublica, where he’ll keep working on data-driven journalism. On one of his few days off between jobs, I talked with him about what it’s like working for a news startup, what he hopes to accomplish at ProPublica, and where he thinks data journalism is headed. Below is a lightly edited transcript. (Disclosure: I used to work at TPM, before Al started there.)

Laura K. McGann: How did you approach your job at TPM? What did you see as your mission there?

Al Shaw: When I started, I came on as an intern right before the ’08 election. At that point, they didn’t have anyone in house who really knew much about programming or design or software. I came on and I saw an opportunity there because TPM is such a breaking-news site, and their whole goal is to do stuff really fast, that they needed someone to do that, but on the technology side, too.

I had a big role in how we covered the 2008 election. We became able to shift the homepage, rearrange stuff. Being able to really elevate what you can do in blogging software. That was kind of the first foray. Then I started redesigning some of the other sections. But the biggest impact I had was redesigning the homepage. That was about a year ago. I had the same goal of being able to empower the editors and nontechnical types to have a bigger palette of what they can do on the site. I created this kind of meta-CMS on top of the CMS that allowed them to rearrange where columns were and make different sections bigger and smaller without having to get into the code. That really changed the way the homepage works.

There is still Movable Type at the core, but there’s a lot of stuff built up around the sides. When we started to build bigger apps, like the Poll Tracker and election apps, we kind of moved off Movable Type all together and started building in Ruby on Rails and Sinatra. They’re hosted on Amazon EC2, which is a cloud provider.

LKM: What have you built that you’re the most proud of?

AS: Probably the Poll Tracker. It was my first project in Rails. It just had enormous success; it now has 14,000 polls in it. Daily Kos and Andrew Sullivan were using it regularly to embed examples of races they wanted to follow and it really has become a central part of TPM and the biggest poll aggregator on the web now. I worked with an amazing Flash developer, Michiko Swiggs, she did the visual parts of the graph in Flash. I think a lot of it was really new in the way you could manipulate the graph — if you wanted to take out certain pollsters, certain candidates, methods, like telephone or Internet, and then you could see the way the trend lines move. You can embed those custom versions.

I think the election tool was also a huge success [too], both technologically and on the design and journalism side. We got linked to from Daring Fireball. We also got linked to from ReadWriteWeb and a lot of more newsy sites. Andrew Sullivan said it was the best place to watch the elections. Because we took that leap and said we’re not going to use Flash, we got a lot of attention from the technology community. And we got a lot of attention from kind of the more political community because of how useable and engaging the site was. It was kind of a double whammy on that.

LKM: What was your experience working with reporters in the newsroom? TPM is turning ten years old, but it’s still got more of a startup feel than a traditional newspaper.

AS: It’s definitely a startup. I would fade in and out of the newsroom. Sometimes I’d be working on infrastructure projects that dealt with the greater site design or something with the ad side, or something beyond the day-to-day news. But then I’d work with the reporters and editors quite a bit when there was a special project that involved breaking news.

So for example, for the Colbert-Stewart rallies we put up a special Twitter wire where our reporters go out to the rallies and send in tweets and the tweets would get piped into a special wire and they’d go right onto the homepage. I worked with editors on how that wire should feel and how it should work and how reporters should interact with it. I remember one concern was, what if someone accidentally tweets somethng and it ends up on the homepage. How do we delete that? I came up with this system with command hashtags, so a reporter could send in a tweet with a special code on it which would delete a certain tweet and no one else would know about that, except for the reporter.

A lot of the job was figuring out what reporters and editors wanted to do and figuring out how to enable that with the technology we had and with the resources we had.

LKM: I remember an instance in my old newsroom where we had a tweet go up on the front page of another site and the frantic emails trying to get it taken down.

AS: Twitter is such an interesting medium because it’s so immediate, but it’s also permanent. We’re having a lot of fun with it, but we’re still learning how best to do it. We did this thing called multi-wire during the midterms, which was a combination of tweets and blog posts in one stream. There was a lot of experimentation with: When do we tweet as compared to a blog post? Should we restrict it to certain hours? That was a really interesting experiment.

LKM: What emerging trends do you see going on in data-driven or interactive journalism?

AS: It’s really good that a lot of sites are starting to experiment more with data-driven journalism, especially as web frameworks and cheap cloud hosting become more prevalent and you can learn Rails and Django, it’s really easy to get a site up that’s based around data you can collect. I do see two kind of disturbing trends that are also happening. One is the rise of infographics. They may not be as useful as they are pretty. You see that a lot just all over the place now. The other problem you see is the complete opposite of that where you’ll get just a table of data filling up your whole screen. The solution is somewhere in between that. You have a better way of getting into it.

It’s really great that there’s kind of a community forming around people that are both journalists and programmers. There’s this great group called Hacks/Hackers that brings those two cohorts together and lets them learn from each other.

LKM: How about at ProPublica? You mentioned you aren’t sure entirely what you’re going to do, but broadly, what do you hope to accomplish there?

AS: I’m most excited about working more closely with journalists on data sets and finding the best ways of presenting those and turning them into applications. That was one thing I was able to do with Poll Tracker, but it didn’t seem like TPM had as big of a commitment to individual stories that could have side applications. Poll Tracker was more of a long-running project. ProPublica is really into delving deeply onto one subject and finding data that can be turned into an application so the story isn’t just a block of text, there’s another way of getting at it.

One of the other things they’re working on is more tools for crowdsourcing and cultivating sources. I know that they want to start building an app or a series of apps around that. And they’re doing some cool stuff with Amazon Mechanical Turk for kind of normalizing and collecting data. I’m sure there’s going to be a lot more fun stuff to do like that.

Older posts are this way If this message doesn't go away, click anywhere on the page to continue loading posts.
Could not load more posts
Maybe Soup is currently being updated? I'll try again automatically in a few seconds...
Just a second, loading more posts...
You've reached the end.

Don't be the product, buy the product!

Schweinderl