Tumblelog by Soup.io
Newer posts are loading.
You are at the newest post.
Click here to check if anything new just came in.

April 20 2012

13:27

Catalysts of Collaboration: What Motivates New Journalism Partnerships

The shift from competition to collaboration in the American newsroom has been so profound that in 2009 the Columbia Journalism Review published an article on "Journalism's collaborative future," arguing that "there is something fundamental under way." That same year, Guardian Editor Alan Rusbridger wrote, "I've seen the future, and it's mutual." The trend is clear, and by all accounts collaborations are expanding and maturing, but do we have a clear enough understanding of what motivates these collaborative efforts? What are the factors inside and outside the newsroom that are inspiring this great collaborative shift?

At MIT's Center for Civic Media in 2010 Scott Rosenberg, executive editor of Grist and Salon co-founder, commented:

There is a professional transition in the field from an environment where competition was the dominant mode of interacting with other organizations to an era where dividing labor and sharing might serve the public better.

The past few years have created a perfect storm of economic crisis, technological transition, and cultural change that have combined to help inspire many journalists to explore news partnerships. Below, I explore three factors that are motivating journalists to work together.

Rapid Technological Change

Journalism practice has always been tied to technological development. In their book, "Four Theories of the Press," Fred Siebert, Theodore Peterson and Wilbur Schramm argue, "The press always takes on the form and coloration of the social and political structures within which it operates." Historically, we've seen this as the telegraph led to the development of the inverted pyramid, the telephone begat the phone interview, and the always-on cable news channels resulted in the 24-hour news cycle.

telegraph.jpg

One thing that differentiates the current batch of technological changes and their impact on journalism is the profound pace of that change. Now, the Internet, mobile devices and new digital tools are prompting the profession of journalism to become more collaborative, by fostering interaction with the public and with other news organizations.

Platforms like Publish2, a content-management system; Stroome, a browser-based video editing platform; and DocumentCloud, a repository for primary documents -- among many others -- are helping to lower the costs of reporting and publishing and connecting individual journalists and newsrooms around shared resources. One of the most ambitious of these projects is the Public Media Platform, which former NPR CEO Vivian Schiller said would "allow all of the content from [various public media] entities -- whether news or cultural products -- to flow freely among the partners and member stations, and, ultimately, also to other publishers, other not-for-profits and software developers who will invent wonderful new products that we can't even imagine."

In addition, collaboration between newsrooms and the public is growing. Examples include CNN's iReport, Huffington Post's OffTheBus and various crowdsourcing projects from ProPublica and others. As a society we are witnessing a technologically driven resurgence in all kinds of sharing, and journalism organizations are a key part of that development.

Economic Factors

This new era of collaboration is not just a function of shiny new gadgets, platforms or programs. It's impossible to ignore the effect the economic recession has had in prompting collaboration. We're living through one of the most difficult periods in the history of the news business (albeit, one of the most exciting), where sharp budget reductions, shrinking ad revenues, dramatic shifts in audiences' media consumption habits, and a range of self-inflicted wounds (from media consolidation to unhealthy debt loads) have upended news organizations' longstanding business models and sparked an age of reinvention and experimentation.

nothiring.jpg

Indeed, many collaborative journalism projects have either been started or are staffed by some of the 30,000-plus people who lost newsroom jobs over the last four years. Esther Kaplan of the Nation Investigative Fund, which "incubates and supports" investigative stories and journalists until the stories are published across a network of magazines like the Atlantic and Mother Jones, has called her effort a "social safety net" for laid-off reporters.

Journalism collaborations present opportunities to share resources and costs, allowing media outlets -- especially independent ones -- to maximize their dwindling budgets. Examples include the Investigative News Network and the Media Consortium, which help independent news organizations with things like back office support, fundraising, and the facilitation of editorial collaborations. In its big-picture report "The Big Thaw," the Media Consortium suggests that the rise of collaboration represents a shift toward a human-centered "alternative economy" that puts community impact and engagement at the center of journalism.

Finally, the economics of collaboration are not only driven by what has been lost, but also by what has been gained as foundations focus on expanding their impact by supporting collaborative projects across organizations. According to J-Lab at American University, foundations have spent upwards of $143 million since 2005 to support new journalism projects, many with collaborative elements. The Corporation for Public Broadcasting (CPB) provided $1 million in mid-2010 for the new Public Media Platform initiative that hopes to create a shared API for community and public media. CPB and the Knight Foundation have also funded regional collaborative journalism ventures between local public TV and radio stations around the country.

Better Journalism

Not all of the factors driving collaboration are external to the work of journalism itself. Many early converts to collaborative journalism argue that it produces a superior product. Spot.Us founder David Cohn has said, on more than one occasion, that if content is king, collaboration is queen. Through collaboration you can tap into skills and expertise outside your organization (such as multimedia production), uncover new story angles, bring in diverse perspectives, and extend the reach and influence of your work.

In the Columbia Journalism Review editorial mentioned above, the editors write:

From foreign capitals to U.S. statehouses, [journalism collaboration] is a way to extend our shrinking newsrooms, begin to rebuild public trust and ensure that the standards of the professional press help shape the development of new journalistic endeavors.

awards.jpg

In recent years, collaborative journalism projects have been earning significant awards. ProPublica has won numerous awards for its collaborations with NPR and other new organizations. The California HealthCare Foundation Center for Health Reporting and the Chico Enterprise-Record won an award from the Association of Health Care Journalists for their joint reporting on woodstove smoke pollution. And the Tiziano Project won a community collaboration award from the Online News Association for its work promoting collaborative journalism in Iraq.

While the other factors above provide external pressure on journalists, most wouldn't embrace collaboration if it wasn't helping them do better journalism.

From Safety in Numbers to Strength in Numbers

Regardless of the catalyst for collaboration, there is a growing sense in the news business that we are all in this together.

The magnitude of this shift toward working together, not just across newsrooms but across the profession as a whole, is perhaps best epitomized by the widespread adoption of a "Show Your Work" ethos. The credo, which encourages journalists and programmers to be transparent with the work they do and share the lessons of their work with the field, was first promoted by the Chicago Tribune apps team last year, and has also been embraced by ProPublica -- but the mantra has spread well beyond these two newsrooms. Dan Sinker, head of the Knight-Mozilla News Technology Partnership, called Show Your Work "perhaps the biggest thing to affect journalism development" of 2011.

Show Your Work is a great example of how collaboration can turn safety in numbers into strength in numbers. Instead of collaborating simply because everyone around you is trying to do more with less, this approach suggests that by working together, we can all achieve more with more. We can build on each other's work, failures and successes to help build better journalism together.

What other motivations and external factors drive journalism collaborations, and how does understanding these catalysts help us better facilitate news partnerships?

Matt Schafer contributed additional research and reporting for this post.

Keep up with all the new content on Collaboration Central by following our Twitter feed @CollabCentral or subscribing to our RSS feed or email newsletter:







Get Collaboration Central via Email

Josh Stearns is a journalist, organizer and community strategist. He is Journalism and Public Media Campaign Director for Free Press, a national, non-partisan, non-profit organization working to reform the media through education, organizing and advocacy. He was a co-author of "Saving the News: Toward a national journalism strategy," "Outsourcing the News: How covert consolidation is destroying newsrooms and circumventing media ownership rules," and "On the Chopping Block: State budget battles and the future of public media." Find him on Twitter at @jcstearns.

Photos above by Flickr users Chris Willis, Bart Heird, and Rob n' Renee.

This is a summary. Visit our site for the full post ».

March 28 2012

15:34

Top-3 tools for recognizing text inside scans

Scans of hard copy documents often require time-consuming analysis with human eyeballs. But optical character recognition can help, and we've identified three great options to get you started. Read More »

December 28 2011

15:20

Idea Lab: Year in Review 2011

2011 year small.jpg

It's been an eventful year on MediaShift's Idea Lab, marked by mergers, beta releases and site redesigns for the many innovators in digital media. This past year also saw the Knight Foundation announce 16 winners of its News Challenge contest, up from 12 grantees in 2010 -- and the total prize money hit $4.7 million, thanks in part to a $1 million contribution from Google.

A couple of themes that ran big among the winners this year were data and mobile. We saw the rise of the hacker-journalist, and many projects were focused on making sense of the stream of data -- think PANDA, ScraperWiki, OpenBlock Rural, Overview, SwiftRiver and DocumentCloud.

awesomefoundation.png

We also saw new interpretations of journalism, such as NextDrop, a mobile platform that helps people in India find out when water is available; Poderopedia, a crowdsourced database that visualizes the relationships among Chile's elite; and the Awesome Foundation, which not only has an awesome name, but is using mini-grants to give others a chance to start up projects of their own.

Here's a look back at just some of the highlights on Idea Lab in 2011.

Just out of beta

Several Knight News Challenge winners announced considerable strides in their projects. The PANDA project, which aims to make basic data analysis quick and easy for news organizations, pushed out a first, and then a second, alpha, adding a login/registration system, dataset search, and complex query support, among other features. It has also been working to integrate directly with fellow News Challenge winner ScraperWiki. "This is speculative at the moment, but has the potential to make the API useful even to novice developers who might not be entirely comfortable writing shell scripts or cron jobs," explained PANDA's Christopher Groskopf.

editing_hangout.jpg

In December, LocalWiki, a 2010 Knight News Challenge winner, announced the first major release of its new LocalWiki software and launched its first focus community, serving Denton, Texas. The LocalWiki project is an ambitious effort to create community-owned, living information repositories that will provide much-needed context behind the people, places, and events that shape our communities.

In addition, SocMap.com, another 2010 Knight News Challenge winner, launched a "tweets" and "places" features on its site, along with plans to debut "local initiatives," "local questions," and a city-planning game in early 2012. And the Cartoonist, which aims to bring newsgames to the masses, showed off a working prototype of the Cartoonist engine for the first time during a demo day hosted by a Georgia Tech research center.

m&a alive and well

There's been no shortage of examples of innovation on Idea Lab, and innovation can, and did this year, lead to acquisitions. Spot.Us, a journalism crowdfunding project that was launched in November of 2008, announced that it was acquired by the Public Insight Network, which is part of American Public Media. "I hope that as Spot.Us and PIN merge, we can continue to push the boundaries in transparency and participation in the process of journalism so that media organizations can better serve the public," Spot.Us founder David Cohn wrote in a post announcing the acquisition.

And earlier in the year, DocumentCloud announced that it had found a long-term home for its project. The startup, which is a catalog of primary source documents and a tool for annotating, organizing and publishing them on the web, merged operations with Investigative Reporters and Editors (IRE), a non-profit grassroots organization committed to fostering excellence in investigative journalism. "IRE has a long and established history of supporting investigative reporting, and we'll be a proud part of their ongoing work to provide journalists with tools that support their reporting," Amanda Hickman, DocumentCloud's former program director, announced.

hacking away

Thumbnail image for hacktoberfest-circle.jpg

The end of September brought with it a four-day hackathon in Berlin organized by Knight-Mozilla, and bringing together programmers and journalists from all over the world. Dan Sinker, who heads up the Knight-Mozilla News Technology Partnership for Mozilla, wrote about the event, which jokingly became known as "Hacktoberfest," and followed up with some reflections on data journalism and opportunities for learning.

Just weeks later, Zeega participated in WFMU's Radiovision Festival, where creative developers and digital storytellers came together for a day of hacking and coding called "Re-Inventing Radio." At the festival, Zeega shared an ultra-early alpha version of its Zeega editor and three projects for people to experiment with.

Brought to you live

In November, we decided to host a live chat on Twitter on the use of SMS and texting technology by journalists, news organizations, radio shows and more. MobileActive's Melissa Ulbricht and Sean McDonald of FrontlineSMS were two Knight News Challenge winners who participated in the live chat, in an effort to explain how services and projects are using SMS to help connect people to important news and information in communities where Internet access is limited.

MobileActive released its Mobile Media Toolkit earlier this year, which provides how-to guides, wireless tools, and case studies on how mobile phones are being used for reporting, news broadcasting, and citizen media.

mobilemedia.png

awards and accolades

A key lesson learned this year was that bigger doesn't necessarily mean better when it comes to new media. The Tiziano Project beat out both CNN and NPR at the 2011 Online Journalism Awards, taking home the Community Collaboration award for its project 360 Kurdistan -- an immersive, nonlinear platform for exploring the culture of the region from the perspectives of both local and professional journalists.

The 2011 award from the Knight Foundation will help the Tiziano Project further develop the 360 technology into a scalable platform that other organizations can use, according to Jon Vidar, the project's executive director. "We will then curate these future 360s on an interactive map and develop a communication layer that will sit on top, allowing visitors to participate in a universal dialog with our students," he wrote in a post.

And November saw Knight-Mozilla announce its 2011/12 News Technology fellows. ScraperWiki's Nicola Hughes and Dan Schultz, a 2007 Knight News Challenge winner and tech wizard extraordinaire for our MediaShift and Idea Lab sites, were two of the innovators who were selected to participate in helping newsrooms around the world develop prototypes for digitally delivering news and information.

No doubt there will be more fantastic innovations and awards to come in 2012! We're looking forward to sharing them with you here on Idea Lab.

July 18 2011

06:16

"Dear Mr (Gil) Brown" - The New York Times and Scotland Yard correspondence

New York Times :: The New York Times has published its request for information about the phone-hacking scandal at The News of the World that was sent last year to Scotland Yard, and its response via DocumentCloud. DocumentCloud is a service which helps journalists to process and publish primary source documents and to make them accessible for a general public via embeddable viewer technology. 

Watch the primary source docs online at www.nytimes.com

July 05 2011

18:30

With its newest round of Knight funding, DocCloud will figure out how to scale reader annotations

Two years ago, DocumentCloud received $719,500 from the Knight News Challenge to build a tool that news organizations could use to upload, share, and then collaboratively read and analyze documents. Since then, the project has not only made good on its promise to “turn documents into data,” introducing its tool into the workflows of investigative reporters in newsrooms across the country, but it’s also found a way to ensure the tool’s sustainment, at least into the near future: In early June, DocCloud’s staff announced that the project would merge operations with, appropriately enough, Investigative Reporters and Editors. And more good news for DocCloud came later in June, when Knight made it a rare double-winner in its News Challenge, granting the project $320,000 to develop an additional feature: reader annotations.

The idea for an annotation mechanism actually originated, indirectly, with the Online News Association, says Amanda Hickman, DocCloud’s program director. She and the team had been thinking about how to include readers more broadly in the process of collaborative document-parsing; at last year’s ONA conference, Hickman began talking with the Public Insight Network‘s Andrew Haeg about how DocCloud might integrate what it already had — an interface that allows a small group of registered users to upload documents and make annotations — with what Public Insight has been up to: finding, and then tracking, a large group of potential story sources. Their conversations, Hickman told me in an email, resulted in “a very whittled down tool for identifying specific experts and asking them to review specific documents, pre-publication.”

That tool is currently in place in the DocCloud system — with the important caveat that, to use it, you have to be invited by a newsroom to do so. From the news organization’s perspective, “there’s a practical limit to who you can invite,” Hickman points out, and “there’s a certain degree of trust involved,” since a public document means, also, public annotations. “It’s a tool that makes sense,” she notes, “if you’re dealing with a few people who aren’t part of your newsroom who need to look over a document.”

It’s a tool that makes less sense, though, if you want broader public input in a given document — which, increasingly, news orgs do. So the reader annotations project faces a tricky task: taking the interplay between expertise and trust that has worked so well in the invite-only annotations system…and building it, somehow, to scale.

And that’s where the Knight funding comes in. With it, DocCloud will figure out how, exactly, to build out the tool’s existing efficiencies to facilitate, and encourage, broader public participation. The goal is pretty much the same as it was when Hickman and Haeg first chatted: to marry DocCloud’s existing annotations infrastructure with the Public Insight approach that helps newsrooms to connect with more sources, more diverse sources, and untapped sources of expertise.

An added twist, though: Whatever system the DocCloud team builds will likely need to interface with outlets’ existing comments infrastructures. Which is both practical and problematic. “We’re not here to reinvent anyone’s moderation system,” Hickman noted in a phone call, “so we’ll have to sort out how to let newsrooms moderate reader annotations,” she says — in basically the same way they already moderate comments. They’ll have to build flexibility, in other words, into a single system to accommodate different outlets’ different approaches to reader commentary.

And they’ll also have to figure out a UI that leverages both the (hoped for) abundance of contributions and the (definite) need for operational efficiency. Visual and otherwise. “If one or two reporters annotate a document, they can make their own decision about how cluttered or uncluttered a page should be,” Hickman points out; with reader annotations, on the other hand, “there’s going to have to be some way to access an uncluttered page if you just want to read the document.”

And that necessity will only expand as the tool’s document set does — especially since a document whose content is meaningful in one way, at one point in time, might take on an entirely different relevance later on, in a different context. So DocCloud will be tasked in part with “figuring out how you present a document that’s annotated in a different context,” Hickman notes. “It’s a really interesting puzzle.”

May 11 2011

15:40

Much Ado About Obama's Birth Certificate on DocumentCloud

As we watched traffic stats skyrocket last month as newsroom after newsroom uploaded President Obama's birth certificate to DocumentCloud and then embedded it, my reaction was hardly one of joy.

Why on Earth is a birth certificate more interesting than, say, the pages and pages of receipts documenting some outrageous meals (15 steaks, two orders of fish and a lamb chop -- for five people submitted by National Grid to the Long Island Power Authority after their Hurricane Earl cleanup)?

I like to think these are the documents we built DocumentCloud for -- that we're here to give a leg up to reporters scrutinizing spurious spending reports (reporting that prompted a formal state investigation) or documenting patent dishonesty and the unusual lengths one California town went to in order to conceal extraordinary salaries paid to city officials.

Vote of Confidence

obama birth certificate.jpg

Forgive me if I was underwhelmed by all the attention that the birth certificate got. My esteemed colleagues, however, helped me see the bright side of the flurry. For one thing, it was fast. Within minutes, 10 different newsrooms had uploaded the birth certificate and embedded it.

That says a lot: It says that when they have something they know their readers want to see, reporters turn to DocumentCloud. That's a huge vote of confidence in us. Plus, we didn't falter under the weight of the tenfold increase in traffic -- that's solid architecture for you. We built DocumentCloud with the hope that we could improve the way newsrooms share source documents with their readers, and at that, we're thrilled to be succeeding.

Increasingly, DocumentCloud is a resource for breaking news. When the news broke that Osama bin Laden had been killed in a town called Abbottabad, a search for "Abbottabad" turned up some pretty rich stuff, most notably that a former Gitmo detainee led U.S. authorities to the Pakistani town back in 2008.

New Feature Roundup

Meanwhile, we're still listening to our users and looking for more ways to make DocumentCloud easier to use and to help reporters give their readers the documents behind the story.

We're looking forward to seeing what our users do with our new tool that lets you embed a single annotation, and we're excited to watch the great uses newsrooms have put document sets to.

From embedding documents accumulated over two decades spent covering an Oregon commune where things went horribly awry to sharing the documents detailing the Federal Reserve's support for ailing financial institutions, or the background material from coverage of a profoundly embarrassed local philanthropist, reporters seem to be getting the hang of embedding document sets.

So we have a question for the reporters who have been using DocumentCloud already: What would have made this even easier for you?

March 31 2011

19:01

March 01 2011

15:07

DocumentCloud Passes Major Milestone: 1 Million Pages Uploaded

DocumentCloud's Jeremy Ashkenas collaborated on this post.

It has been less than a year since DocumentCloud began adding users to our beta. Late Monday morning, a user uploaded our millionth page of primary source documents.

The thousands of documents in our catalog have arrived in small batches: five pages here, twenty there. The vast majority of the 65,000 documents that those million pages comprise remain private, but we're fast closing in on 10,000 public documents in our catalog.

Broad Appeal

Journalists are using DocumentCloud to publish all sorts of documents, including these:

Remaking History

Documents in our catalog reach back into the past, as well. In 1970 Ruben Salazar was killed by police while covering an anti-war protest in east Los Angeles. A story rife with controversy, questions, and suspicions, his death became a rallying point in the Mexican American civil rights movement. Forty years later -- after refusing a public records request for documents that might shed some light on the circumstances of his death -- the Los Angeles County Sheriff's Department agreed to turn the files over to the Office of Independent Review.

While Los Angeles Times reporters waited for the report, they assembled their own folio of early clippings on Ruben Salazar. Readers can review FBI files obtained by the Times in 1999 and LAPD records on the department's repeated clashes with the journalist as well as a draft of the report prepared by the Office of Independent Review.

Join the Cloud

You can browse recently published documents by searching for "filter: published" or read up on other searches you might want to run. Here's hoping that the next year brings millions more pages, and more great document-driven reporting.

February 23 2011

19:55

3 Ways to Expand the News Ecosystem

Spot.Us founder David Cohn has convened a virtual carnival: he's posing monthly questions that he'd like to see journalists take a stab at answering. The latest: how do we diversify the news ecosystem? He put it differently -- "Considering your unique circumstances, what steps can be taken to increase the number of news sources?" -- but I'm pretty sure the end goal is a greater diversity of information and expanded news ecosystem.

What can I do, personally? I can use my technical skills to make document-based investigative reporting a little easier and a little more transparent. But "you knew I was going to say that (because of the work I do for DocumentCloud.

1. Push for More Public Data

And every journalist and citizen can push for increased access to public information. That would, for example, making it possible for more New Yorkers to cover New York City. It doesn't take much to publish public data reliably, it just takes some political will.

2. Increase Collaboration

Another thing we can do is increase story collaboration. No one newsroom can ever reveal the complete picture. The full story become clear when many reporters come at an issue, each from their own unique perspective. If some of those reporters have gone to journalism school and have been mentored by a prize winning journalist and others are just calling it like they see it without even the benefit of a copy editor, more power to us all. (And if you imagine that the former never get a story outrageously wrong or that the latter are never downright spot-on, you haven't been paying attention.)

One reporter, working alone to cover the statehouse, is never going to get as much done as 10 reporters, each actively trying to sniff out a corruption case that hasn't already been discovered.

In the process, though, some journalists have developed a nasty habit of pretending they've got a scoop when, in fact, they're re-telling a story first uncovered by a neighborhood blog. One of my favorite hyper-local bloggers, who regularly reports on her precinct community meetings and other things nearly no other news outlet has the resources to cover, also keeps an unfortunate running tally of stories of hers that were picked up by the press without so much as a nod.

3. Share the Credit

Which brings me to my final point: Share the credit. It really is okay for journalists to look for local leads in neighborhood blogs. But when a reporter finds one, she should be sure to find a way to weave a tip-o-the-hat into her narration of the story.

Don't pretend you work in a vacuum. Giving credit is common courtesy. And it leaves your friendly local bloggers free to be incensed by horrendous construction gaffes and intransigent municipal bureaucracies instead of ticked off at you.

Journalists should strive to share their reporting and pool their technical skills and give one another the courtesy of due credit. Those simple steps would go a long way toward increasing the size, scope, and vitality of the news ecosystem.

December 22 2010

15:00

Amy Webb: The IPv4 problem, geofencing, and lots of hyperlocal

Editor’s Note: We’re wrapping up 2010 by asking some of the smartest people in journalism what the new year will bring.

Here’s digital media consultant Amy Webb of Webbmedia Group, on hyperlocal startups, tablets, geofencing, and more.

Every device that connects to the Internet, from mobile phones to MiFis to computers to TiVOs, needs a unique ID number (also called an IP address) in order to make contact with other devices on the network. The world will run out of addresses by March 2011. This means that for those in developing areas like China and India who finally have access to technology, they won’t be able to get online. But it also means that large-scale U.S. providers such as Comcast won’t be able to support new customers as they have in the past. Why? Our current standard, IPv4, is the Internet Protocol developed in 1981. It’s been 30 years, and we’re out of numbers. The next iteration is IPv6, which is ultimately more secure and is much more extensible. Eventually, ISPs will have to make the switch and migrate all of their customers. However, those people connecting via IPv6 won’t be able to access content that’s being housed on IPv4. The New York Times, Washington Post, CNN, NPR, local blogs — basically any content producer who hopes to continue reaching a worldwide audience — will either have to start migration now or will face losing millions of visitors starting Q3 next year.

Lots of new hyperlocal initiatives will launch before summer 2011 by a vast number of traditional media organizations. Millions and millions of dollars will be spent recreating templated sites based on zip code or geography alone. All of the local ad dollars being counted on will instead shift towards social commerce sites like LivingSocial and Groupon, which have started to include compelling editorial content. Interest among journalists will grow, while consumer interest continues to stagnate. Only the hyper-personal sites that focus on niche content and geography rather than neighborhoods alone will succeed.

2011 will be the year of the tablet. We’ll see close to two dozen tablets come to market, most running some version of Android. Consumers will continue to love the iPad, while publishers will continue developing what is essentially a web-centric experience for a device that does much, much more. Smart entrepreneurs will leapfrog traditional news organizations by focusing on dynamic content curation via algorithm. Think Pulse 2.0, Flipboard, Wavii — but even more engaging.

Geofencing will become an integral part of the checkin experience in 2011. Right now, many mobile social networks use a fuzzy radius to locate members, and it’s easy to game the system. But it’s also harder for retailers and others interested in social commerce to effectively use networks like Foursquare and Gowalla because it’s difficult to verify that a user is actually inside of a store or at a specific location. For news orgs trying to syndicate content, the best many can do now is to leave vague tips around town. Geofencing technology requires very strict location parameters, allowing a number of interesting possibilities. For example, check-ins can be triggered automatically, expiring assets (such as event tickets or breaking news alerts) can be pushed to users, and a moving target — like a parade or car chase — can be tracked or commented on. And with geofencing, someone can’t check into his favorite restaurant repeatedly while driving past it his way to work.

Data-filled firehoses will spring leaks everywhere in 2011. And not just WikiLeaks. Twitter is releasing a personal metrics dashboard soon. Other social networks are discussing how to release data streams about and for their users and the content being discussed. News organizations will soon find a fantastic opportunity to harness all of that data, to parse it, and to develop stories about everything from the U.S. government to our cultural zeitgeist. DocumentCloud is a breakthrough, an essential tool developed by journalists for journalists. I hope to see more of its ilk released in 2011.

December 21 2010

17:00

At #Niemanleaks, a new generation of tools to manage floods of new data

Whether it’s 250,000 State Department cables or the massive spending databases on Recovery.gov, the trend in data has definitely become “more.” That presents journalists with a new problem: How do you understand and explain data when it comes by the gigabyte? At the Nieman Foundation’s one-day conference on secrecy and journalism, presenters from the New York Times, Sunlight Foundation, and more offered solutions — or at least new ways of thinking about the problems.

Think like a scientist

With the massive amounts of primary documents now available, journalists have new opportunities to bring their readers into their investigations — which can lead to better journalism. John Bohannon, a contributing correspondent for Science Magazine, said his background as a scientist was great preparation for investigative reporting. “The best kind of investigative journalism is like the best kind of science,” he said. “You as the investigator don’t ask your readers to take your claims at face value: You give them the evidence you’ve gathered along the way and ask them to look it with you.”

It’s not a radical idea, but it’s one being embraced in new ways. For Bohannon, it meant embedding with a unit in Afganistan and methodically gathering first-hand data about civilian deaths — a more direct and reliable indicator than the less expensive and safer method of counting media-reported deaths. He also found his scientific approach was met with more open answers from a military known for tight information control. “Sometimes if you politely ask for information, large powerful organizations will actually give it to you,” he said.

The future will be distributed: BitTorrent, not Napster

Two of the projects discussed, Basetrack and DocumentCloud, invite broader participation in the news process, the former in the sourcing and the latter with the distribution.

Basetrack, a Knight News Challenge winner, goes beyond the normal embedding process to more actively involving the Marines of First Battalion, Eighth Marine Regiment as they deploy overseas in reporting their experiences. Teru Kuwayama, who leads the project and deployed with the battalion to Afghanistan, said ensuring that confidential information wasn’t released, putting lives in danger, was essential to building trust and openness with the project. So Basetrack built a “Denial of Information” tool that allowed easy, pre-publication redactions, with the caveat that the fact of those redactions — and the reasons given for them — would be made public. It’s a compromise that promises a greater intimacy and a collaborative look at life at war while ensuring the safety of the soldiers.

Fellow News Challenge winner DocumentCloud, on the other hand, distributes the primary documents dug up through traditional investigative journalism, such as historical confidential informant files or flawed electoral ballot designs. Aron Pilhofer, editor of interactive news at The New York Times, said he was unsure about whether journalists would actually use it when his team began working on the project — but since then dozens of organizations have embraced it, happy to take readers along for the ride of the investigative process.

These new ways of distributing reporting were just the beginning, Pilhofer said, with a trend that will likely push today’s marquee whistleblower out of the limelight. “WikiLeaks was very much a funnel going in and very much a funnel going out,” he said. “Distributed is the future.” A new project, called OpenLeaks, will embrace a less centralized model, building technology to allow anonymous leaks without a central organization to be taken out.

Big data’s day is here

The panel also tackled how to digest truly massive data sets. Bill Allison, editorial director of the Sunlight Foundation, detailed how his organization collected information on everything from earmarks to political fundraising parties. Allison said making this data actually meaningful required context, which could be simple as mapping already available data or scoring government databases based on understandable criteria.

“We try to make the information easy to use,” he said. But beyond the audience of curious constituents who use Sunlight’s tools, a much broader audience is reached as hundreds of journalists around the country use Sunlight’s tools to dig up local stories they might not otherwise have noticed — creating a rippling effect of transparency

15:00

Tracking documents, numbers, and military social media: New tools for reporting in an age of swarming data

To conclude our series of videos from the Nieman Foundation’s secrecy and journalism conference, here’s a video of the day’s final session — the Labbiest of the bunch. Our own Megan Garber moderates a set of presentations on new digital approaches to dealing with new data and new sources.

The presenters: John Bohannon, contributing correspondent for Science Magazine; Teru Kuwayama, Knight News Challenge winner for Basetrack; Aron Pilhofer, editor of interactive news at The New York Times and Knight News Challenge winner for DocumentCloud; and Bill Allison, editorial director of the Sunlight Foundation. Below is an embed of the session’s liveblog.

December 09 2010

17:03

Altering Docs? Now There's a Tool for That in DocumentCloud

When we embarked on the DocumnetCloud project, tools for altering documents were the furthest thing from our minds. After all, a responsible journalist doesn't tweak source documents!

But one of the first papers to embed material using DocumentCloud needed to do just that. The Chicago Tribune accompanied their coverage of a troubled foster home with a collection of letters and court orders. Though the documents offered an excellent illustration of the state child services agency's lax oversight and slipped follow-ups, they were predictably full of personal information about children in the foster care system, individual agency staff names and other personal and identifying details about private individuals that the Tribune opted to omit from their reporting. That decision, however, left the news apps team replacing the whole stack of letters multiple times before the package was finally ready to post.

A tool, right inside of DocumentCloud, for replacing, removing and re-ordering the pages of a document would have helped them a lot.

When the "PBS NewsHour" shared a century old hand-written Mark Twain essay, our OCR tools were not nearly up to the task of reading his handwriting. NewsHour transcribed the 10-page essay by hand and we worked with them to replace the text stored in DocumentCloud and displayed on the embedded letters.

By the time that Memphis' Commercial Appeal wanted to run a lengthy series of handwritten letters from Abdulhakim Mujahid Muhammad, a young Memphis-born man who opened fire on a military recruiting center in Little Rock last summer, we at DocumentCloud were busy supporting nearly 200 newsrooms -- offering to hide the text tab was the best we could do.

What NewsHour and Commercial Appeal really needed was a tool, right inside of DocumentCloud, with which to edit the text of each document.

And so, we've assembled what we think is a sweet suite of tools to let you re-order pages, insert new ones, delete old ones and edit the text that will appear in your embedded document. Check out our user guide to see how it all works. We welcome your bugs, feedback, rants, raves and, as ever, your documents.

November 24 2010

17:01

Last Minute News Challenge Tips: Tell a Story, Be Realistic, and More

Planning to spend the long weekend finalizing your Knight News Challenge application? It's too late for my favorite bit of advice ("don't wait until the last minute!")m, but as someone who's been involved with three different winning projects, I like to fancy that I've got got some insight into what makes a good project.

A half dozen prospective applicants have sat down with me to workshop their News Challenge ideas, and I think I've helped them think through their projects to get them to a more viable place. The application process isn't hard, but you do need to give some sincere thought to your project or you're just wasting your time. Here's the advice I keep giving people:

Tips for the News Challenge

Focus on work you really want to do -- If you have a great idea but aren't really personally invested in making it happen, you're going to face a long, long slog if it gets funded. Three different people have complained to me that software developers put a ton of time and energy into developing Knight proposals that didn't wind up getting funded. That's always a let down, but it shouldn't be the end of the world. If you do make it past the first round, Knight is going to ask you a lot of hard questions and work with you to revise your proposal. If you don't get funded, you're left with a pretty solid and well thought out proposal that you can shop around if you really want to raise the money you need to get funded. That's a good thing!

Tell a solid story of engagement -- I'm not an expert on what is and isn't news, and I cut my teeth most recently at Gotham Gazette, which has pretty distinct standards for what qualifies. The most fascinating story won't find a home there if it doesn't have any apparent policy implications. I'm pretty sure that Knight doesn't look for policy implications alone, but if you can't tell me a solid story that takes me from your project to citizens (and non citizens!) and helps make them more engaged in decision making in their communities in some tangible way, something is missing from your project. That, or you're making me work far too hard to understand why this matters. So spell it out.

Make a realistic budget -- Grants awarded by the news challenge vary wildly in amount. Meaning: You should be honest with yourself about what it will cost to see your project through. A low ball request could leave you without enough money to finish what you started, and could be a sign to Knight's reviewers that you don't have a good understanding of what your project is going to take. A stratospheric budget isn't any more realistic.

Have a realistic outreach plan -- If you've got a great idea but no idea how to connect with the users that should be taking advantage of it, you've got a silo. Think this will be useful to a community? Go out and talk to people about what you're trying to do, how you think it will help and listen to what they say about how they want to use it. Not just what they think would be nice for other people to use, but what they want and will use.

Proofread! Proofread! Proofread! -- Your goal is to impress people, and to impress upon them that they should take a chance on your bright idea. Attention to grammatical details doesn't matter to everyone, but to some people a misplaced modifier is like nails on a chalkboard. Why risk alienating a reviewer?

Don't give up -- Knight's reviewers are going to look at zillions of proposals. If you're convinced that yours is a good idea but Knight turns it down, don't just quit. Keep looking for ways to make it happen, and keep listening to your community for insights that might make it a stronger project next time.

PS. I really have been involved with three Challenge winners. I wasn't around to help write the proposal, but I joined Gotham Gazette full time as director of technology after they won a 2007 news challenge grant to develop a series of games about public policy. Two years later, I helped develop Gotham Gazette's winning Councilpedia proposal before I joined the DocumentCloud team.

October 27 2010

18:07

DocumentCloud Users Make Ballot Design An Election Issue

When we make lists of the kinds of source documents users can upload to DocumentCloud, they can get pretty long. DocumentCloud is court filings, hearing transcripts, testimony, legislation, lab reports, memos, meeting minutes, correspondence. I can say with absolute confidence that in all of our planning, "ballots" never once came up as the sort of document a news organization might want to annotate for readers. Our relentlessly creative users have shown us otherwise.

This summer, the Memphis Commercial Appeal rounded out its guide to August's primary elections with a sample ballot. Their digital content editor told us that many readers who'd missed the sample ballot in the print edition turned to the version online as primary day approached. Earlier this month, they added the general election ballot to that guide.

New York Ballots

WNYC, New York City's NPR affiliate, also published a few ballots this summer. In an effort to comply with a 2002 federal law that mandates significant updates to voting systems in each state, New York City introduced paper ballots for the 2010 primary election, replacing the city's famously arcane voting machines. One look at the new design and everyone was up in arms, proclaiming its absurdity, but WNYC actually invited a group of ballot design experts to review the city's new ballots. Their findings: the ballot was confusing

Design for Democracy works to increase civic participation, in part through a ballot design project that aims to make voting easier and more accurate. WNYC used Design for Democracy's feedback to annotate a sample ballot on their blog, offering readers vital voting advice.

When the city released sample ballots for November's general election, a local think tank pointed out that the instructions erroneously advise voters to mark the oval above their candidate's name. In fact, the relevant ovals appear below candidate's names. WNYC highlighted the issue by embedding a sample ballot on their blog. Apparently the "oval above" language was mandated by state law. Don't believe me? See for yourself -- WNYC posted the legislation, with the relevant passage highlighted.

From now on, my laundry list of things DocumentCloud catalogs will most definitely include ballots.

October 21 2010

16:00

MuckRock makes FOIA requests easy, but will reporters use it?

Making freedom of information requests can be a daunting task. If it’s not an agency dragging its heels on releasing documents or asking for a fee large enough to buy a compact car, then it’s the actual process of the, well, process. You’ve got to identify the right agency, contact the right administrator, find out whether they take requests in the mail or electronically, and even then you’ve got to word your request precisely or risk ending up with liquor licenses when you wanted restaurant inspections.

It’s a system begging for simplification. While DocumentCloud is making it easier to wrangle and make sense of public records, MuckRock wants to make FOIA requests similarly effortless.

On its face MuckRock is a tool that allows journalists of all stripes (pro to amateur and in between) to make document requests easier. Think of it like Netflix: You tell MuckRock what you’re looking for and it provides suggestions, ultimately getting you what you’re looking for. But instead of Starship Troopers, you wind up with a nicely formatted request letter to your record agency of choice.

But in keeping with the idea of transparency, MuckRock also provides an online tracking service to see the progress of requests, and acts as a repository for all the records collected through the site.

“The idea we had was making a nice, modern interface for the end user on a very backwards, outdated, finicky process,” Michael Morisy, MuckRock’s co-founder told me.

The question, perhaps a bigger one than going from an analog system to a digital one, is whether journalists (particularly investigative ones known for being careful with their records) are willing to trade control over information (and potentially their scoop) for a streamlined, simplified FOIA process.

“The real tragedy is in a lot of cases a reporter will get hundreds of pages of government documents and they might use two or three sentences from them or might not use them at all,” he said. “And then they go into some filing room for all eternity where they’re lost.”

Morisy and co-founder Mitchell Kotler built a FOIA wizard of sorts that takes users through the steps of selecting federal or local agencies and the particular data they’re interested in. Among the options are areas like budgets, public contracts, sex offender lists and pet license information. From there MuckRock acts as an emissary, sending the request letter and providing the tracking tool to show whether the documents are being processed or are past due. Though the site currently only lists state and local data for Massachusetts, Morisy said they are hoping to expand. But the local level may be where MuckRock could have the most impact, making it easier for newspapers or local sites to create projects like a public employee salary database.

The idea was to build a service that anyone could use — a long-time journalist, a neighborhood blogger, or someone simply looking to get answers out of city hall. The value to bloggers or citizen journalists seems clear: Providing not just tools but guidance on the sometimes labyrinthian process of making document requests. But for journalists working at established media outlets, the pitch is a little more tricky. “We’ve kind of found our sweet spot right now is helping out anybody who’s at least that pro-am journalist or a community blogger,” said Morisy, who has written for the New York Daily News and Business 2.0. “But also the overburdened reporter who writes stories, blogs, tweets, and has to juggle investigations.” In theory, a new resource to help newsrooms expedite FOIA requests would be a help, particularly at a time when shrinking staff and rising demand on reporters may exclude investigative projects. In reality, experienced journalists are generally more comfortable undertaking FOIA requests themselves, if not for accuracy than to keep a story under wraps from competitors or the government itself.

“We want to give people as much control over what’s public and what’s private — the last thing we ever want to do is ruin a reporter’s scoop,” Morisy said. With that in mind, MuckRock now allows users to embargo their requests and documents for up to 30 days after receiving a response from an agency.

DocumentCloud, WikiLeaks, and attempts at crowdsourced document analysis show that technology has enabled better methods of obtaining, displaying and dissecting information. What may also need to adapt, Morisy argues, is the mindset around collecting and reporting on public documents. Journalists often have to commit to a balancing act, asking for documents only to keep them hidden away during their reporting, Morisy said. MuckRock encourages transparency by its design, Morisy said, and he hopes it encourages all journalists to make as much as their reporting open as possible.

“I think it’s important for people to see how these documents are obtained, and that it’s not just reporters that have access to documents — anyone can get them,” Morisy said.

September 07 2010

19:55

DocumentCloud Helps Newspapers Bring Transparency to Government

Since we last updated readers on DocumentCloud's progress, we've made it much easier to upload a lot of documents at once, and introduced a related documents search that uses data about names and places provided by OpenCalais to find documents that are probably related to the one you're looking at. We've also added a bit more contextto the data we help reporters comb through. Most of this work is happening inside the gates of the DocumentCloud workspace, but it is resulting in some lively reporting. For example...

Using Documents to Tell the Story

This summer, as the federal 5th Circuit Court of Appeals prepared to hear arguments in a challenge to the University of Texas's affirmative action policy, Texas Tribune complemented its coverage of the case with nearly 200 pages of annotated court documents, including the original district court ruling, the university's appellate brief, as well as that of the plaintiffs in the case.

The Las Vegas Sun incorporated quite a trove of documents into its series on hospital care in Las Vegas. Readers were invited to browse everything from Department of Health and Human Services reports to individual records, right along with the Sun's reporters. When they say that hospital-acquired infections cost the country $30 billion per year or account for close to 100,000 deaths, they back each number up with original documents.

The Columbia Missourian annotated the city budget and took a local blogger to task for exaggerating Columbia, Missouri's cash reserves.

When Texas Governor Rick Perry challenged reporters to find anyone who can out-work him, Texas Tribune posted the governor's May 2010 schedule alongside that of Florida's Gov. Crist, New York's Gov. Paterson and California's Gov. Schwarzenegger and invited readers to help them skim over a hundred pages of briefings, receptions and photo ops for stories deserving of a closer look.

The Washington Post supplemented its reporting on the cozy relationship between the oil industry and the federal agency assigned to regulate them with an annotated report on the prospects for "Moving beyond Conflict" between regulator and regulated. Their document cache also included reports outlining just how cozy things had gotten by 2008. As Emily Keller pointed out in Free Government Info, a transparency project, documents like these give more transparency to journalism itself.

New Features in the Testing Lab

We're also hard at work fine tuning the document viewer, transforming it into something that users could reasonably plug into a template with a narrower content column. Thus far folks have been stuck with a full page viewer. We haven't fully rolled it out yet, but we've worked with a couple of our beta testers to implement it already.

Iowa State has a new men's basketball coach, and the Des Moines Register included all 14 pages of his contract to their coverage of the finer points contained in it. Among the unusual clauses? Hoiberg can walk away if the university decides to increase academic standards for student athletes beyond the NCAA's minimum.

Meanwhile, at the Santa Fe Reporter, Alexa Schirtzinger opted not to publish tables of information right inside her story on elder abuse in New Mexico, but she did use her staff blog to share the data that she had such a hard time tracking down. An annotation highlights the numbers that showed her that New Mexico fields more abuse complaints per nursing home bed than any other state.

DocumentCloud watchers will notice that they posted the contract right on the same page as Randy Peterson's writeup instead of displaying the document in a full page. We'll be making tweaks like this a lot easier for all of our users. In the meantime, if you're skilled at the art of reverse engineering JavaScript, you can view the source of the Register's story (or the Reporter's) to see just how they toggled the sidebar or zoom on those documents.

August 03 2010

17:51

DocumentCloud Helps Arizona Paper with Annotated Immigration Law

We opened the DocumentCloud floodgates less than six months ago and we're still working hard to make DocumentCloud a better tool. We're rolling out improvements at a healthy clip including SSL support, better documentation, and support for cross-newsroom collaboration. We continue to listen to feedback from our really incredible crop of beta testers (who now number close to 500!).

There are nearly 100 newsrooms participating in the DocumentCloud beta and requests are still pouring in. We've been doing a fair amount of outreach and more is in the works, but it turns out that our users are our best advocates: After John Addams in Great Falls, Montana, blogged about his experiences with DocumentCloud we were deluged with requests from Montana news organizations large and small.

Use Cases in Arizona, Chicago, Memphis

The really great stories about how reporters are using DocumentCloud continue to surprise all of us.

Not long after Arizona's governor signed that state's now infamous immigration law, the Arizona Republic published the bill in full, complete with annotations by a local law professor. Republic reporters told us that traffic to the annotated legislation outpaced the paper's popular entertainment guide in its first weekend, and continues to draw traffic as the bill stays in the news.

Meanwhile, in Chicago, reporters at the Tribune have been uploading each document and transcript entered into evidence in former governor Rod Blagojevich's corruption trial -- the documents are just part of their extensive coverage of the trial.

In Memphis, the Commercial Appeal published a sample ballot alongside their voter guide.

These are just a few of the great uses reporters have put DocumentCloud to -- there are many more great stories already out there and plenty of new ones on the way.

May 04 2010

16:00

Why does the BBC want to send its readers away? The value of linking

The BBC aims to double the number outbound clicks from its site by 2013. That’s double the number of people sent away from the BBC site — intentionally. In a recent BBC blog post, BBC News website editor Steve Herrmann cites a BBC strategy review document which lays out the goal of

Turning the site into a window on the web by providing at least one external link on every page and doubling monthly ‘click-throughs’ to external sites.

One external link per page will seem laughably low to any seasoned blogger, but intentionally increasing outbound traffic is positively radical for a mainstream newsroom. It’s a goal that might baffle proponents of the walled garden approach to web sites, or raise howls of protest among those who feel that aggregators are parasites, but Herrmann wrote that the BBC sees it as a service to its readers:

Related links matter: They are part of the value you add to your story — take them seriously and do them well; always provide the link to the source of your story when you can; if you mention or quote other publications, newspapers, websites — link to them.

This comes in the wake of £600 million in cuts to the BBC budget, about 15 percent of the huge organization’s spending. That includes a 25-percent cut to the BBC website’s budget, which will halve the number of top-level sections by 2013. The BBC has also delayed its iPad/iPhone news reading application in the U.K. after industry complaints that it is crowding private newsrooms out of the market. (American users can already use the iPad app.)

Is the BBC’s plan to increase external links an enlightened editorial policy, or is this just spin on a downsizing announcement? Are they aiming to provide a valuable curation service to their readers, have they been forced by regulators to reduce the scope of their work, or is this really a cash-strapped move towards a cheaper, aggregator-style news organization? I asked Herrmann to explain.

He told me by email that

The strategy envisages the BBC as a cultural and public space, one that isn’t trying to sell anything and can be trusted. It sets out the aim of building this broader public space by working with other public cultural organisations to share and promote a wider range of content.

So the principle for BBC Online, which covers news, weather, sport and programme content, is that it should be “a window on the web”, guiding audiences to the best of the internet as well as partnering with external providers — and that is why we want to increase the click-throughs.

Nonetheless, he acknowledged that competitive concerns played some role in the decision. “We do need to leave space for others,” he wrote.

The move is also about transparency. In an age where many source documents are available in electronic form online, there’s often little reason that readers shouldn’t have access to the same material that reporters use to write their stories. Yet the practice of showing your sources is still less then common among many news organizations. I asked Herrmann if the BBC had a specific policy on source linking.

This is something else I have raised in the blog. There should be a principle that we do link to the most relevant and useful information, including the source documents, wherever we can. That’s not something new — we’ve always had huge interest from users in the source documents we make available for government budget announcements, for example — but it is a restatement of the principle, and a signal of our intent to try to do this as well as we possibly can. Also, as I have started to discuss in the blog post, there is some devil in the detail — for example sometimes the source document isn’t online at time of writing, or it is behind a paywall, or requires subscription — so we are thinking these things through. I’m interested in trying to formulate and develop the best policy with the help of the detailed feedback we are getting from our users.

There is a lively discussion around the details of an ideal source linking policy in the comments to Herrmann’s post, especially as regards academic journals and other non-free sources. It’s also worth mentioning the DocumentCloud project, a serious attempt to build a journalistic document repository which solves some of these problems, such as keeping documents private before publication.

But does the courtesy of linking extend to your competition? “Do what you do best and link to the rest” has become a new-media maxim, but mainstream news organizations are still loathe to send readers to someone else’s reporting. So does the BBC intend to link more often to stories produced by other news sources?

Yes, news organisations and other sources. That is the focus of my recent blog post. We are in the process of working out what this means for our day-to-day working practices on the newsdesk, how to link more but also better. We’ve had links on stories since we started, and we have long had an automated module that pulls in related stories from other news sites, but how can technology help us to do this even better, and what does the journalist working on a story need to change in the way they approach what they do?

Aggregators flourish because users find them useful. The weekly link roundup and the top-ten list remain perennial blogging forms. And while every statement in news writing is supposed be attributed, in practice Wikipedia articles link to their sources far more reliably than news stories. The BBC may be on to something here.

April 15 2010

16:28
Older posts are this way If this message doesn't go away, click anywhere on the page to continue loading posts.
Could not load more posts
Maybe Soup is currently being updated? I'll try again automatically in a few seconds...
Just a second, loading more posts...
You've reached the end.

Don't be the product, buy the product!

Schweinderl