Tumblelog by Soup.io
Newer posts are loading.
You are at the newest post.
Click here to check if anything new just came in.

November 10 2010

18:30

Talking Points Memo’s first developer talks startup life, jumping to ProPublica and data journalism

What’s it like being the only in-house techie at a news startup? Talking Points Memo’s first developer Al Shaw says “it’s kind of like being a reporter….you have to be a generalist,” doing everything from ad-side work to election-night interactives.

Shaw was the primary technical force behind most of the bells and whistles that cropped up at TPM over the past two years, including a redesign that lets producers switch up the layout of the homepage, and an array of slick interactives like the real-time election results tracker that made TPM look a lot less like a scrappy startup and more like an establishment outlet on Election Night earlier this month. (Shaw is quick to explain he had some help on the election map from Erik Hinton, TPM’s technical fellow.) He’s also been good about blogging about his technical endeavors in ways that could be useful to his peers at other news organizations.

Shaw announced last month he is leaving TPM to start a new gig at ProPublica, where he’ll keep working on data-driven journalism. On one of his few days off between jobs, I talked with him about what it’s like working for a news startup, what he hopes to accomplish at ProPublica, and where he thinks data journalism is headed. Below is a lightly edited transcript. (Disclosure: I used to work at TPM, before Al started there.)

Laura K. McGann: How did you approach your job at TPM? What did you see as your mission there?

Al Shaw: When I started, I came on as an intern right before the ’08 election. At that point, they didn’t have anyone in house who really knew much about programming or design or software. I came on and I saw an opportunity there because TPM is such a breaking-news site, and their whole goal is to do stuff really fast, that they needed someone to do that, but on the technology side, too.

I had a big role in how we covered the 2008 election. We became able to shift the homepage, rearrange stuff. Being able to really elevate what you can do in blogging software. That was kind of the first foray. Then I started redesigning some of the other sections. But the biggest impact I had was redesigning the homepage. That was about a year ago. I had the same goal of being able to empower the editors and nontechnical types to have a bigger palette of what they can do on the site. I created this kind of meta-CMS on top of the CMS that allowed them to rearrange where columns were and make different sections bigger and smaller without having to get into the code. That really changed the way the homepage works.

There is still Movable Type at the core, but there’s a lot of stuff built up around the sides. When we started to build bigger apps, like the Poll Tracker and election apps, we kind of moved off Movable Type all together and started building in Ruby on Rails and Sinatra. They’re hosted on Amazon EC2, which is a cloud provider.

LKM: What have you built that you’re the most proud of?

AS: Probably the Poll Tracker. It was my first project in Rails. It just had enormous success; it now has 14,000 polls in it. Daily Kos and Andrew Sullivan were using it regularly to embed examples of races they wanted to follow and it really has become a central part of TPM and the biggest poll aggregator on the web now. I worked with an amazing Flash developer, Michiko Swiggs, she did the visual parts of the graph in Flash. I think a lot of it was really new in the way you could manipulate the graph — if you wanted to take out certain pollsters, certain candidates, methods, like telephone or Internet, and then you could see the way the trend lines move. You can embed those custom versions.

I think the election tool was also a huge success [too], both technologically and on the design and journalism side. We got linked to from Daring Fireball. We also got linked to from ReadWriteWeb and a lot of more newsy sites. Andrew Sullivan said it was the best place to watch the elections. Because we took that leap and said we’re not going to use Flash, we got a lot of attention from the technology community. And we got a lot of attention from kind of the more political community because of how useable and engaging the site was. It was kind of a double whammy on that.

LKM: What was your experience working with reporters in the newsroom? TPM is turning ten years old, but it’s still got more of a startup feel than a traditional newspaper.

AS: It’s definitely a startup. I would fade in and out of the newsroom. Sometimes I’d be working on infrastructure projects that dealt with the greater site design or something with the ad side, or something beyond the day-to-day news. But then I’d work with the reporters and editors quite a bit when there was a special project that involved breaking news.

So for example, for the Colbert-Stewart rallies we put up a special Twitter wire where our reporters go out to the rallies and send in tweets and the tweets would get piped into a special wire and they’d go right onto the homepage. I worked with editors on how that wire should feel and how it should work and how reporters should interact with it. I remember one concern was, what if someone accidentally tweets somethng and it ends up on the homepage. How do we delete that? I came up with this system with command hashtags, so a reporter could send in a tweet with a special code on it which would delete a certain tweet and no one else would know about that, except for the reporter.

A lot of the job was figuring out what reporters and editors wanted to do and figuring out how to enable that with the technology we had and with the resources we had.

LKM: I remember an instance in my old newsroom where we had a tweet go up on the front page of another site and the frantic emails trying to get it taken down.

AS: Twitter is such an interesting medium because it’s so immediate, but it’s also permanent. We’re having a lot of fun with it, but we’re still learning how best to do it. We did this thing called multi-wire during the midterms, which was a combination of tweets and blog posts in one stream. There was a lot of experimentation with: When do we tweet as compared to a blog post? Should we restrict it to certain hours? That was a really interesting experiment.

LKM: What emerging trends do you see going on in data-driven or interactive journalism?

AS: It’s really good that a lot of sites are starting to experiment more with data-driven journalism, especially as web frameworks and cheap cloud hosting become more prevalent and you can learn Rails and Django, it’s really easy to get a site up that’s based around data you can collect. I do see two kind of disturbing trends that are also happening. One is the rise of infographics. They may not be as useful as they are pretty. You see that a lot just all over the place now. The other problem you see is the complete opposite of that where you’ll get just a table of data filling up your whole screen. The solution is somewhere in between that. You have a better way of getting into it.

It’s really great that there’s kind of a community forming around people that are both journalists and programmers. There’s this great group called Hacks/Hackers that brings those two cohorts together and lets them learn from each other.

LKM: How about at ProPublica? You mentioned you aren’t sure entirely what you’re going to do, but broadly, what do you hope to accomplish there?

AS: I’m most excited about working more closely with journalists on data sets and finding the best ways of presenting those and turning them into applications. That was one thing I was able to do with Poll Tracker, but it didn’t seem like TPM had as big of a commitment to individual stories that could have side applications. Poll Tracker was more of a long-running project. ProPublica is really into delving deeply onto one subject and finding data that can be turned into an application so the story isn’t just a block of text, there’s another way of getting at it.

One of the other things they’re working on is more tools for crowdsourcing and cultivating sources. I know that they want to start building an app or a series of apps around that. And they’re doing some cool stuff with Amazon Mechanical Turk for kind of normalizing and collecting data. I’m sure there’s going to be a lot more fun stuff to do like that.

June 04 2010

18:23

How a Test Suite Can Help Your Open Source Project Grow

At CityCircles, we've been fortunate to work with a local developer who is passionate about our project's goal of developing hyper-local communication tools for mass audiences. Our first implementation of that is a platform for light rail passengers in Phoenix, Arizona.

That said, one person can't carry the entire load, especially as the project inevitably evolves from its humble beginnings and wire frames.

One solution that's worth considering is sinking some funds into a test suite -- a closed environment where other developers who share a vision for the project can develop new features with the approval of the "master" developer. This is the approach we recently took with CityCircles.

Test Suite

In March, we contracted with a local development shop called Integrum Technologies to build a test suite. The project is connected to our code base and includes simulated tasks that other developers can build toward and "test." If these features pass muster in the test suite, then we can push those changes to our code base permanently. If they do not, then the developer can tweak them until they do without ruining the live site.

The test suite took almost three weeks to build and cost us roughly $9,500. (That may seem pricey to some, but good Ruby on Rails developers are not cheap. In our case, Integrum specializes in test suites.) However, for startups, this is a very helpful option for reaching goals of new features and functions on a budget. Open-source software developers that are looking for a "portfolio" piece and are attracted to the project's mission can participate at a fraction of the cost to the project. In return, they receive publicity and, in some cases, a promise of future paid work. The idea is that everyone wins.

Once your test suite is completed, start poking around your local area for developer meetups. Go online and subscribe to developer forums and Google groups. In our case, the project is built in Ruby on Rails. I have joined the Rails community's leading Google Group with the intent of marketing this test suite to developers.

I've also been invited to attend Integrum's weekly "hacknight" meetup in Chandler, a Phoenix suburb. Tomorrow night, I'll be there to spread the gospel of the project and hope that our handy test suite attracts the right crew.

Use these test suites to your advantage, as simulators like them can also help create an organic "buzz" around the project as well. Include the developers' names on the open-source software license, too. That will also help.

But be mindful of the pitfalls. Just as there are several developers that may want to participate, they may not have the chops to complete the work in a timely or accurate manner. It helps to have a strong master developer to sign off on their work.

May 21 2010

13:00

The programmer majored in English: A fascinating study of the NYT’s Interactive News unit

At the University of Texas’s International Symposium on Online Journalism conference last month, a series of academics presented papers on the future of news. There’s great stuff, including (Lab contributor) Seth Lewis’s analysis of the professional and participatory logic of the Knight News Challenge and (Lab contributor) C.W. Anderson’s argument for a more holistic approach to academic analysis of news structures.

One that we, at least, found particularly compelling: Cindy Royal’s study of the New York Times’ vaunted Interactive Newsroom Technologies unit. (Think of it as the academic, ethnographic version of “The Renegades at the New York Times,” last year’s New York magazine profile of the team.)

Royal, a Texas State University assistant professor who focuses on digital media and culture, spent a week with the team in an effort to “gain a systematic understanding of the role of technology in the ever-changing newsroom, driven by the opportunities and challenges introduced by the Internet.” The resulting paper examined the group of eleven guys (they’ve since added one gal) widely recognized to be the vanguard of the hacker-journalist movement — and put fascinating anecdotal data behind team leader Aron Pilhofer’s insistence that the group’s mandate is editorial as much as technological.

Though the full paper (PDF) is well worth a read, here’s the slide deck:

One of Royal’s more intriguing findings: Many members of the team don’t have traditional education in programming. “Undergraduate degrees were varied in Art & Design, Anthropology, English, History, Urban Planning, Rhetoric and of course, Journalism. Only two had done extensive educational preparation in a computer-oriented field, and another two had received technical-oriented minors in support of liberal arts degrees.” And their hacking skills? Largely self-taught. “Most had either taken up computing on their own at a very young age or had gravitated toward it due to necessity for a specific job.”

The core unifying quality Royal found among the staff wasn’t a specific programming skill or even a set of those skills. It was passion. Curiosity. Enjoyment of the work and openness to new processes and approaches. “More than half our team members didn’t know Ruby on Rails [one of the Times' core web framework technologies] before they started here,” one member notes. (Team member commentary throughout Royal’s paper is anonymous.) “It’s really more about the concepts inherent in the language,” says another.

However: The editorial part — “getting” the journalism — is also key. (“When I was hired, they definitely cared about how much I was interested in journalism and what my ideas were for projects.”) As is the collaboration part — the institutional realization of the open-source-centric approach the team takes toward its work. The department, Royal notes, “was founded to reduce bureaucracy and introduce flexibility in the process of creating each project, so the group could react more like a reporting team than a support organization.” That’s a goal that the Times is still actively pursuing — most recently, of course, with its decision to move Jill Abramson from her managing editor post to allow her to focus intensively, if temporarily, on digital operations. Abramson will likely be spending at least some of that time in the same way Royal did: studying and learning from the paper’s Interactive News unit.

“The culture of technology is different than that of journalism,” Royal notes. “They each carry different ideas about objectivity, transparency, sharing of information and performance. By merging these cultures, what emerges in terms of a hybrid dynamic? How do the actors, their backgrounds and training, their processes and the organizational structure affect the products they deliver?”

“The Journalist as Programmer” builds on the work of academics like Michael Schudson and Dan Berkowitz, taking an ethnographic (and, more broadly, sociological) approach to news systems — under the logic, as Royal writes, that “news products and ultimate change are not the result of one force or set of forces, but a complete system that encompasses the organization, individual actors and the culture that surrounds them.”

As she explained it to me: “I just wanted to learn about the processes, and who these people were. I knew that they had to be a unique department with unique skills and backgrounds. Because the average programmer really doesn’t have much interest in traditional journalism or storytelling. And the average journalist doesn’t know a lot about programming. So who are these hybrid people — and where did they come from, and how did they learn this stuff?”

After all, “programming and data is journalism,” Royal says. “And it can be practiced in such a way that it can create interaction, user engagement, and more information in terms of seeking the truth. Especially when you talk about Freedom of Information access to government data — if the public can have access to that in a way that makes sense to them, or in a way that’s easy for them to use, then that’s just really powerful.”

December 01 2009

02:54

Looking for a Web Developer for Nonprofit Project

Meaningful Media is a non-profit organization dedicated to increasing the quality, visibility and impact of social issue media.

We are redesigning our website and need a nonprofit-friendly web developer to bring our vision to life.

Project Details:

read more

Older posts are this way If this message doesn't go away, click anywhere on the page to continue loading posts.
Could not load more posts
Maybe Soup is currently being updated? I'll try again automatically in a few seconds...
Just a second, loading more posts...
You've reached the end.

Don't be the product, buy the product!

Schweinderl