Tumblelog by Soup.io
Newer posts are loading.
You are at the newest post.
Click here to check if anything new just came in.

May 30 2013

01:35

The Social Media Editor is Dead, Long Live the Social Media Editor!

The social media editor isn't dead, but the Newsroom Social Media Rockstar Ninja Guru's days are numbered. And thank goodness for that. [...]

August 20 2012

13:34

How Wikipedia Manages Sources for Breaking News

Almost a year ago, I was hired by Ushahidi to work as an ethnographic researcher on a project to understand how Wikipedians managed sources during breaking news events.

Ushahidi cares a great deal about this kind of work because of a new project called SwiftRiver that seeks to collect and enable the collaborative curation of streams of data from the real-time web about a particular issue or event. If another Haiti earthquake happened, for example, would there be a way for us to filter out the irrelevant, the misinformation, and build a stream of relevant, meaningful and accurate content about what was happening for those who needed it? And on Wikipedia's side, could the same tools be used to help editors curate a stream of relevant sources as a team rather than individuals?

pakistan.png

Ranking sources

When we first started thinking about the problem of filtering the web, we naturally thought of a ranking system that would rank sources according to their reliability or veracity. The algorithm would consider a variety of variables involved in determining accuracy, as well as whether sources have been chosen, voted up or down by users in the past, and eventually be able to suggest sources according to the subject at hand. My job would be to determine what those variables are -- i.e., what were editors looking at when deciding whether or not to use a source?

I started the research by talking to as many people as possible. Originally I was expecting that I would be able to conduct 10 to 20 interviews as the focus of the research, finding out how those editors went about managing sources individually and collaboratively. The initial interviews enabled me to hone my interview guide. One of my key informants urged me to ask questions about sources not cited as well as those cited, leading me to one of the key findings of the report (that the citation is often not the actual source of information and is often provided in order to appease editors who may complain about sources located outside the accepted Western media sphere). But I soon realized that the editors with whom I spoke came from such a wide variety of experience, work areas and subjects that I needed to restrict my focus to a particular article in order to get a comprehensive picture of how editors were working. I chose a 2011 Egyptian revolution article on Wikipedia because I wanted a globally relevant breaking news event that would have editors from different parts of the world working together on an issue with local expertise located in a language other than English.

Using Kathy Charmaz's grounded theory method, I chose to focus editing activity (in the form of talk pages, edits, statistics and interviews with editors) from January 25, 2011 when the article was first created (within hours of the first protests in Tahrir Square), to February 12 when Mubarak resigned and the article changed its name from "2011 Egyptian protests" to "2011 Egyptian revolution." After reviewing the big-picture analyses of the article using Wikipedia statistics of top editors, and locations of anonymous editors, etc., I started work with an initial coding of the actions taking place in the text, asking the question, "What is happening here?"

I then developed a more limited codebook using the most frequent/significant codes and proceeded to compare different events with the same code (looking up relevant edits of the article in order to get the full story), and to look for tacit assumptions that the actions left out. I did all of this coding in Evernote because it seemed the easiest (and cheapest) way of importing large amounts of textual and multimedia data from the web, but it wasn't ideal because talk pages, when imported, need to be re-formatted, and I ended up using a single column to code data in the first column since putting each conversation on the talk page in a cell would be too time-consuming.

evernote.png

I then moved to writing a series of thematic notes on what I was seeing, trying to understand, through writing, what the common actions might mean. I finally moved to the report writing, bringing together what I believed were the most salient themes into a description and analysis of what was happening according to the two key questions that the study was trying to ask: How do Wikipedia editors, working together, often geographically distributed and far from where an event is taking place, piece together what is happening on the ground and then present it in a reliable way? And how could this process be improved?

Key variables

Ethnography Matters has a great post by Tricia Wang that talks about how ethnographers contribute (often invisible) value to organizations by showing what shouldn't be built, rather than necessarily improving a product that already has a host of assumptions built into it.

And so it was with this research project that I realized early on that a ranking system conceptualized this way would be inappropriate -- for the single reason that along with characteristics for determining whether a source is accurate or not (such as whether the author has a history of presenting accurate news article), a number of important variables are independent of the source itself. On Wikipedia, these include variables such as the number of secondary sources in the article (Wikipedia policy calls for editors to use a majority of secondary sources), whether the article is based on a breaking news story (in which case the majority of sources might have to be primary, eyewitness sources), or whether the source is notable in the context of the article. (Misinformation can also be relevant if it is widely reported and significant to the course of events as Judith Miller's New York Times stories were for the Iraq War.)

nyt.png

This means that you could have an algorithm for determining how accurate the source has been in the past, but whether you make use of the source or not depends on factors relevant to the context of the article that have little to do with the reliability of the source itself.

Another key finding recommending against source ranking is that Wikipedia's authority originates from its requirement that each potentially disputed phrase is backed up by reliable sources that can be checked by readers, whereas source ranking necessarily requires that the calculation be invisible in order to prevent gaming. It is already a source of potential weakness that Wikipedia citations are not the original source of information (since editors often choose citations that will be deemed more acceptable to other editors) so further hiding how sources are chosen would disrupt this important value.

On the other hand, having editors provide a rationale behind the choice of particular sources, as well as showing the variety of sources rather than those chosen because of loading time constraints may be useful -- especially since these discussions do often take place on talk pages but are practically invisible because they are difficult to find.

Wikipedians' editorial methods

Analyzing the talk pages of the 2011 Egyptian revolution article case study enabled me to understand how Wikipedia editors set about the task of discovering, choosing, verifying, summarizing, adding information and editing the article. It became clear through the rather painstaking study of hundreds of talk pages that editors were:

  1. storing discovered articles either using their own editor domains by putting relevant articles into categories or by alerting other editors to breaking news on the talk page,
  2. choosing sources by finding at least two independent sources that corroborated what was being reported but then removing some of the citations as the page became too heavy to load,
  3. verifying sources by finding sources to corroborate what was being reported, by checking what the summarized sources contained, and/or by waiting to see whether other sources corroborated what was being reported,
  4. summarizing by taking screenshots of videos and inserting captions (for multimedia) or by choosing the most important events of each day for a growing timeline (for text),
  5. adding text to the article by choosing how to reflect the source within the article's categories and providing citation information, and
  6. editing disputing the way that editors reflected information from various sources and replacing primary sources with secondary sources over time.

It was important to discover the work process that editors were following because any tool that assisted with source management would have to accord as closely as possible with the way that editors like to do things on Wikipedia. Since the process is managed by volunteers and because volunteers decide which tools to use, this becomes really critical to the acceptance of new tools.

sources.png

Recommendations

After developing a typology of sources and isolating different types of Wikipedia source work, I made two sets of recommendations as follows:

  1. The first would be for designers to experiment with exposing variables that are important for determining the relevance and reliability of individual sources as well as the reliability of the article as a whole.
  2. The second would be to provide a trail of documentation by replicating the work process that editors follow (somewhat haphazardly at the moment) so that each source is provided with an independent space for exposition and verification, and so that editors can collect breaking news sources collectively.

variables.png

Regarding a ranking system for sources, I'd argue that a descriptive repository of major media sources from different countries would be incredibly beneficial, but that a system for determining which sources are ranked highest according to usage would yield really limited results. (We know, for example, that the BBC is the most used source on Wikipedia by a high margin, but that doesn't necessarily help editors in choosing a source for a breaking news story.) Exposing the variables used to determine relevancy (rather than adding them up in invisible amounts to come up with a magical number) and showing the progression of sources over time offers some opportunities for innovation. But this requires developers to think out of the box in terms of what sources (beyond static texts) look like, where such sources and expertise are located, and how trust is garnered in the age of Twitter. The full report provides details of the recommendations and the findings and will be available soon.

Just the beginning

This is my first comprehensive ethnographic project, and one of the things I've noticed when doing other design and research projects using different methodologies is that, although the process can seem painstaking and it can prove difficult to articulate the hundreds of small observations into findings that are actionable and meaningful to designers, getting close to the experience of editors is extremely valuable work that is rare in Wikipedia research. I realize now that in the past when I actually studied an article in detail, I knew very little about how Wikipedia works in practice. And this is only the beginning!

Heather Ford is a budding ethnographer who studies how online communities get together to learn, play and deliberate. She currently works for Ushahidi and is studying how online communities like Wikipedia work together to verify information collected from the web and how new technology might be designed to help them do this better. Heather recently graduated from the UC Berkeley iSchool where she studied the social life of information in schools, educational privacy and Africans on Wikipedia. She is a former Wikimedia Foundation Advisory Board member and the former Executive Director of iCommons - an international organization started by Creative Commons to connect the open education, access to knowledge, free software, open access publishing and free culture communities around the world. She was a co-founder of Creative Commons South Africa and of the South African nonprofit, The African Commons Project as well as a community-building initiative called the GeekRetreat - bringing together South Africa's top web entrepreneurs to talk about how to make the local Internet better. At night she dreams about writing books and finding time to draw.

This article also appeared at Ushahidi.com and Ethnography Matters. Get the full report at Scribd.com.

July 19 2011

19:15

Wikipedia Taps College 'Ambassadors' to Broaden Editor Base

From what I can tell, most of my fellow educators spend more time criticizing Wikipedia than engaging with it.

The conversation tends to go round in a fairly tiresome circle: The first educator points to an article on the subject of his/her expertise and points to a glaring error to demonstrate that the whole enterprise is worthless. The interlocutor responds with a (highly debated) study to argue that "Wikipedia is more accurate than Encyclopedia Britannica."

wikipediachart.jpg

But neither side comes to terms with the real Wikipedia revolution: It represents a restructuring of the architecture of knowledge. In the decade since its founding, the crowdsourced platform has grown exponentially, radically improved its content, and established a firm foothold in the online environment, now ranking as the fifth most-visited site in the world. The entire enterprise is based on Wikipedia's utopian vision, as spelled out on the back of the staff business cards: "Imagine a world in which every single human being can freely share in the sum of all knowledge."

Campus Ambassadors

That said, many observers -- starting with the Wikimedia Foundation itself -- realize that this vision is far from realized. This has led the foundation to launch a series of initiatives designed to improve the infrastructure and broaden participation. One of the most intriguing developments is the Public Policy Initiative and its corps of campus ambassadors.

The challenges are formidable. Let's leave aside, for the moment, the two-thirds of the world's population that has yet to gain access to the Internet. The creation of Wikipedia content has striking limitations, even among the 400 million users who visit the site every month. According to Wikipedia's own estimates, only 0.02-0.03 percent of visitors actively contribute to articles.

And although technically, content can be created by anyone with an online account, the pattern of participation is admittedly skewed. According to Barry Newstead, the foundation's chief global development officer, "Eighty percent of our page views are from the Global North, and 83 percent of our edits." The English language Wikipedia's content and participation far outstrip those for its 270 other languages, especially non-Western. Of the active contributors, between 80 and 85 percent are male, and half are under 22. Furthermore, participation has plateaued (and even declined) over the last few years, settling in at somewhere between 80,000 and 100,000 active editors per month in all languages.

What are the obstacles to growth? First of all, "Wikipedian" culture is known for its contentious behavior, especially toward newcomers who haven't mastered the arcane style and coding. One result is that the content has become skewed toward geek topics, featuring state-of-the-art articles on technology, science fiction and military history, with more erratic offerings in the humanities and social sciences.

straightening a skewed pattern of participation

wikimediafoundationlogo.png

In early July, the Wikimedia Foundation renewed its efforts to improve the balance by holding its first Higher Education Summit as part of the Public Policy Initiative. According to the foundation leadership, the goals of the year-old project are to:

• bring in more quality content in underserved fields, starting with public policy;
• narrow the gender gap by recruiting increased female participation;

• improve diversity of contributors, and

• make the initiation process more user-friendly.

Backed by a new strategic plan and a $1.2 million grant from the Stanton Foundation, the Wikipedia Public Policy Initiative was born. The foundation decided that the natural focal point for the effort was academia. Colleges and universities were, after all, the traditional centers of learning -- and it made sense to look to students who were researching and writing papers as potential contributors of content.

Mentoring Professors and Students

The missing link was the Wikipedia knowledge. This was addressed by the creation of a cohort of Wikipedia "ambassadors" to coach and mentor professors and students through the wickets. Fifty-four campus ambassadors were selected over 2010-2011, charged with offering on-site support in classroom and personal tutorial settings.

These were often students with an extensive (and successful) record of creating and editing Wikipedia content. In other cases, they were university librarians, tech support, and other staff who took on the challenge as part of their classroom support services. (At the summit, Sue Gardner, director of the Wikimedia Foundation, proudly pointed out that almost half of the campus ambassadors are female.) The campus ambassadors were complemented by 91 online ambassadors, experienced Wikipedians who offer support to students in any school.

The United States was divided into 10 regions, each assigned a regional ambassador. Professors from 24 colleges and universities signed up as inaugural Wikipedia Teaching Fellows to participate. In return, the professors made a commitment to assigning Wikipedia content creation as part of their course requirements, and to stage the assignments over the course of the semester, to allow for an editorial learning curve.

By coincidence, I had created an unwitting control group for this effort. Last fall I assigned my Media & Society class at Bard College to write or edit a Wikipedia entry, unaware that there was a Wikimedia program for classroom support. I had a few Wikipedia edits under my belt, but I was unprepared for my students' struggles with Wikipedia policies on issues such as notability, verifiability and sourcing. These policies are highly specific, not always intuitive, and don't necessarily mirror academic practice.

My international students writing on foreign subjects had far more trouble than my U.S. students in publishing their articles, even if they were of comparable quality (partly, I believe, because it's harder to provide approved citations for local information about countries such as Afghanistan and Burma). I was also remiss in not directing my students toward the sandbox to develop their articles before posting them -- leading to some swift and merciless deletions.

For many of us, the Higher Education Summit was a welcome opportunity to meet campus and online ambassadors and to hear how fellow professors worked with the project in the classroom. I was surprised to learn that while some of the professors were experienced Wikipedians, many of them had little editing experience with the platform. This was not regarded as a problem. The program was structured to task the ambassadors with Wikipedia skills, allowing professors to focus on shaping syllabi and course content. (The summit's invitees included professors of law, anthropology, political science, and literature.)

More User-Friendly Editing

whats hot public policy.jpg

At the same time, there are many signs that the Wikimedia Foundation is eager to make the editing process more user-friendly. It has been conducting usability studies to see where the bumps are. Wikipedia has been expanding its live help and a rich trove of learning materials for newbies. These resources are scattered across the Wikipedia terrain and not easy to locate, but the foundation is taking active steps to both build out and codify the materials. It's also sponsoring some friendly competition with a leaderboard to monitor which classes are posting the most contributions over the semester, as well as a "What's Hot" list of most edited articles by students.

The Wikimedia Foundation states that over the next five years it hopes to increase the number of readers to a billion, and the percentage of editors in the Global South to 37 percent. The international initiative is starting with Brazil, India and the Middle East/North Africa, which have already begun to receive advance guards of campus and online ambassadors. (The summit included academics from Brazil, India, Germany, the U.K. and Canada, as well as the U.S.)

The Wikimedia Foundation's Newstead told the summit attendees that the organization is still struggling with the challenge of adapting to mobile platforms, the bridgehead for online media in much of the world. "At this point you can't edit on mobile; it's read-only," he reported. "Most people who move to mobile stop surfing the web. They just surf apps."

Whatever the challenges, the Wikipedia ambassadors are recruiting a new generation of professors and students to carry the vision forward. There are all kinds of creative challenges to adapting classroom assignments to the mission. Students like publishing their classwork online, but express frustration at team members who don't pull their weight. Professors enjoy the classroom enthusiasm, but struggle with the mechanics of grading collaborative writing projects and articles that are edited by a broader community. Nonetheless, there's every indication that the Wikimedia Foundation's experiment in higher education will take Wikipedia to another stage of its wildly unpredictable ride.

Anne Nelson is an educator, consultant and author in the field of international media strategy. She created and teaches New Media and Development Communications at Columbia's School of International and Public Affairs (SIPA). She consults on media, education and philanthropy for Anthony Knerr & Associates. Her most recent book is Red Orchestra. She tweets as @anelsona, was a 2005 Guggenheim Fellow, and is a member of the Council on Foreign Relations.

This is a summary. Visit our site for the full post ».

September 28 2010

08:54

September 16 2010

18:33

What I read today…

January 07 2010

11:49

AP names manager of Social Networks and News Engagement

Lauren McCullough, who has led The Associated Press’ efforts to engage social networks, has been named manager of Social Networks and News Engagement at the new AP Nerve Center. McCullough will head the Social Network Center, one of four components of the new headquarters center in New York. She will direct the work of editors there and around the company in pursuing journalistic material from social networks, promoting AP’s presence and content on social networks, and providing feedback to news managers on topics of high interest on social networks.

December 22 2009

20:30

KNC 2010: The Journalism Shop offers vetted editorial talent for hire

[EDITOR'S NOTE: We're highlighting a few of the entries in this year's Knight News Challenge, which just closed Tuesday night. Did you know of an entry worth looking at? Email Mac or leave a brief comment on this post. —Josh]

You may have already heard of The Journalism Shop, the assemblage of ex-Los Angeles Times staffers that has evolved into an editorial matchmaking service. (Its survey of ex-LATers detailing their predictions for the paper’s failure got some notice from Romenesko a couple weeks ago.)

It’s an online co-op where former Times reporters, editors, and designers can hang a freelance shingle and land jobs. The site, which evolved out of an email list for laid-off staffers, currently has around 30 members. And it’s throwing its hat into the ring for a Knight News Challenge grant. According to their application, they hope to build:

— a national network of regional reporters/editors/researchers/graphic artists who will create original work on spec, to be placed by The Journalism Shop editors.

— acting as an assigning conduit for editors looking for freelancers (a modern version of the old photo agency structure, but for writers and editors).

— a “pitching engine” to solicit assignments for our members.

— pursuing grants for topic specific journalism.

— building out the existing website to publish those stories (we’re working on some ideas for that now.

Scott Martelle, one of the co-founders and a former Times reporter himself, said The Journalism Shop helps assignment editors quickly find and tap experienced journalists for coverage. And since all of The Journalism Shop’s members are former LAT staffers, they have a built-in credibility with editors that not all freelancers can boast.

“There’s very little going on out there that tries to keep experienced journalists in the profession,” Martelle said. “We’re trying to keep people alive until the Big Bang ends and the solar systems begin to coalesce again.”

As you can see from the Shop’s Facebook page, work doesn’t always originate from traditional news organizations. The boundaries have expanded to include things like alumni magazines, annual reports, consulting, book work, and the like.

A News Challenge grant would allow Martelle and co-founder Brett Levy to dedicate more time to the project. Specifically, they want to build out the infrastructure and create additional opportunities for their stable of writers through outreach and advertising. Martelle said the model could also be extended to other locales.

Older posts are this way If this message doesn't go away, click anywhere on the page to continue loading posts.
Could not load more posts
Maybe Soup is currently being updated? I'll try again automatically in a few seconds...
Just a second, loading more posts...
You've reached the end.

Don't be the product, buy the product!

Schweinderl