Tumblelog by Soup.io
Newer posts are loading.
You are at the newest post.
Click here to check if anything new just came in.

June 11 2013

17:00

OpenData Latinoamérica: Driving the demand side of data and scraping towards transparency

“There’s a saying here, and I’ll translate, because it’s very much how we work,” Miguel Paz said to me over a Skype call from Chile. “But that doesn’t mean that it’s illegal. Here, it’s ‘It’s better to ask forgiveness than to ask permission.””

Paz is a veteran of the digital news business. The saying has to do with his approach to scraping public data from governments that may be slow to share it. He’s also a Knight International Journalism Fellow, the founder of Hacks/Hackers Chile, and a recent Knight News Challenge winner. A few years ago, he founded Poderopedia, a database of Chilean politicians and their many connections to political organizations, government offices, and businesses.

But freeing, organizing, and publishing data in Chile alone is not enough for Paz, which is why his next project, in partnership with Mariano Blejman of Argentina’s Hacks/Hackers network, is aimed at freeing data from across Latin America. Their project is called OpenData Latinoamérica. Paz and Blejman hope to build a centralized home where all regional public data can be stored and shared.

Their mutual connection through Hacks/Hackers is key to the development of OpenData Latinoamérica. The network will make itself, to whatever extent possible, available for trouble shooting and training as the project gets off the ground and civic hackers and media types learn both how to upload data sets as well as make use of the information they find there.

Another key partnership helping make OpenData Latinoamérica possible is with the World Bank Institute’s Global Media Development program, which is run by Craig Hammer. Hammer believes the data age is revolutionizing government, non-government social projects, and how we make decisions about everyday life.

“The question for us, is, What are we gonna do with the data? Data for what? Bridging that space between opening the data and how it translates into improving the quality of people’s lives around the world requires a lot of time and attention,” he says. “That’s really where the World Bank Institute and our programmatic work is focused.”

A model across the Atlantic

Under Hammer, the World Bank helped organize and fund Africa Open Data, a similar project launched by another Knight fellow, Justin Arenstein. “The bank’s own access-to-information policy provides for a really robust opportunity to open its own data,” Hammer says, “and in so doing, provide support to countries across regions to open their own data.”

Africa Open Data is still in beta, but bringing together hackers, journalists, and information in training bootcamps has already led to reform-producing journalism. In a post about the importance of equipping the public for the data age, Hammer tells the story of Irene Choge, a journalist from Kenya who attended a training session hosted by the World Bank in conjunction with Africa Open Data.

She…examined county-level expenditures on education infrastructure — specifically, on the number of toilets per primary school…Funding allocated for children’s toilet facilities had disappeared, resulting in high levels of open defecation (in the same spaces where they played and ate). This increased their risk of contracting cholera, giardiasis, hepatitis, and rotavirus, and accounted for low attendance, in particular among girls, who also had no facilities during their menstruation cycles. The end result: poor student performance on exams…Through Choge’s analysis and story, open data became actionable intelligence. As a result, government is acting: ministry resources are being allocated to correct the toilet deficiency across the most underserved primary schools and to identify the source of the misallocation at the root of the problem.

Hammer calls Africa Open Data a useful “stress test” for OpenData Latinoamérica, but Paz says the database was also a natural next step in a series of frustrations he and Blejman had encountered in their other work.

“Usually, the problem you have is: Everything is cool before the hackathon, and during the hackathon,” says Paz. “But after, it’s like, who are the people who are working on the project? What’s the status of the project? Can I follow the project? Can I be a part of the project?” The solution to this problem ended up being Hackdash, which was actually Blejman’s brainchild — an interface that helps hackers keep abreast of the answers to those questions and thereby shore up the legacy of various projects.

So thinking about ways that international hackers can organize and communicate across the region is nothing new to Paz and Blejman. “One hackathon, we would do something, and another person who didn’t know about that would do something else. So when we saw the Open Data Africa platform, we thought it was a really great idea to do in Latin America,” he says.

Blejman says the contributions of the World Bank have been essential to the founding of OpenData Latinoamérica, especially in organizing the data bootcamps. Hammer says he sees the role of the bank as building a bridge between civic hackers and media. “More than a platform,” he says it’s, “an institution in and of itself to help connect sources of information to government and help transform that data into knowledge and that knowledge into action.”

Giving people the tools to understand the power of data is an important tenet of Hammer’s open data philosophy. He believes the next step for Big Data is global data literacy, which he says is most immediately important for “very specific and arguably strategic public constituencies — journalists, media, civic hackers, and civil society.” Getting institutions, like newspapers, to embrace the importance of data literacy rather than relying on individual interest is just one goal Hammer has in mind.

“I’m not talking about data visualization skills for planet Earth,” he says. “I’m saying, it’s possible — or it should be possible — for anybody that wants to have these skills to have them. If we’re talking about data as the real democratizer — open data as meaningful democratization of information — then it has to be digestible and accessible and consumable by everyone and everybody who wants to access and digest and consume it.”

Increasing the desire of the public for more, freer data is what Hammer calls stoking the demand side. He says it’s great if governments are willingly making information accessible, but for it to be useful, people have to understand its power and seek to unleash it.

“What’s great about OpenData Latinoamérica is it’s in every way a demand-side initiative, where the public is liberating its own data — it’s scraping data, it’s cleaning it,” he says. “Open data is not solely the purview of the government. It’s something that can be inaugurated by public constituencies.”

For example, in Argentina, where the government came late to the open data game, Blejman says he saw a powerful demand for information spring up in hackers and journalists around him. When they saw what other neighboring countries had and what they could do with that information, they demanded the same, and Argentina’s government began to release some of that data.

“We need to think about open data as a service, because no matter how much advocacy from NGOs, people don’t care about ‘open data’” per se, Paz says. “They care about data because it affects their life, in a good or bad way.”

Another advantage Bleman and Paz had when heading into OpenData Latinoamérica was the existence of Junar, a Chilean software platform founded by Javier Pajaro, who was a frustrated analyst when he decided to embrace open data platforms and help others do the same. Blejman said that, while Africa Open Data opted for CKAN, using a local, Spanish-language company that was already familiar to members of the Hacks/Hackers network has strengthened the project, making it easier to troubleshoot problems as they arise. He also said Junar’s ability to give participating organizations more control fit nicely into their hands-off, crowd-managed vision for future day-to-day operation of the database.

Organizing efforts

Paz and Blejman have high hopes for the stories and growth that will come from OpenData Latinoamérica. “What we expect from these events is for people to start using data, encourage newspapers to organize around data themes, and have the central hub for what they want to consume,” Blejman said.

They hope to one day bring in data from every country in Latin America, but they acknowledge that some will be harder to reach than others. “Usually, the federated governments, it’s harder to get standardized data. So, in a country like Argentina, which is a federated state with different authorities on different levels, it’s harder to get standardized data than in a republic where there’s one state and no federated government,” says Paz. “But then again, in Chile, we have a really great open data and open government and transparency allows, but we don’t have great data journalism.” (Chile is a republic.)

Down the road, they’d also like to provide a secure way for anonymous sources to dump data to the site. Paz says in his experience as a news editor, 20–25 percent of scoops come from anonymous tips. But despite developments like The New Yorker’s recent release of Strongbox, OpenData Latinoamérica is still working out a secure method that doesn’t require downloading Tor, but is more secure than email. Blejman also added that, for now, whatever oversight they have over the quality and accuracy of the original data they’re working with is minimal: “At the end, we cannot control the original sources, and we are just trusting the organizations.”

But more than anything, Paz is excited about seeing the beginnings of the stories they’ll be able to tell. He plans to use documents about public purchases made by Chile’s government to build an app that allows citizens to track what their government is spending money on, and what companies are being contracted those dollars.

Another budding story exemplifies the extent to which Paz has taken to heart Craig Hammer’s emphasis on building demand. In Chile, there is currently a significant outcry from students over the rising cost of education. Protests in favor of free education are ongoing. In response, Paz decided to harness this focus, energy, and frustration into a scrape-a-thon (or #scrapaton) to be held June 29 in Santiago. They will focus on scraping data on the owners of universities, companies that contract with universities, and who owns private and subsidized schools.

“There’s a joke that says if you put five gringos — and I don’t mean gringos in a disrespectful way — if you put five U.S. people in a room, they’re probably going to invent a rocket,” says Paz. “If you put five Chileans in a room, they’re probably going to fight each other. So one of the things — we’re not just building tools, we’re also building ways of working together, and making people trust each other.” Blejman added that he hopes the recent release of a Spanish-language version of the Open Data Handbook (El manual de Open Data) will further facilitate collaboration between hackers in various Latin American countries.

With a project of this size and scope, there are also some ambitious designs around measurement. Paz hopes to track how many stories and projects originate with datasets from OpenData Latinoamérica. Craig Hammer wants to quantify the social good of open data, a project he says is already underway via the World Wide Web Foundation’s collaboration with the Open Data for Development Camp.

“If there is a cognizable and evidentiary link between open data and boosting shared prosperity,” Hammer says, “then I think that would be, in many cases, the catalytic moment for open data, and would enable broad recognition of why it’s important and why it’s a worthwhile investment, and broad diffusion of data literacy would really explode.”

Hammer wants people to take ownership of data and realize it can help inform decisions at all levels, even for individuals and families. Once that advantage is made clear to the majority of the population, he says, the demand will kick in, and all kinds of organizations will feel pressured to share their information.

“There’s this visceral sense that data is important, and that it’s good. There’s recognition that opening information and making it broadly accessible is in and of itself a global public good. But it doesn’t stop there, right? That’s not the end,” he says. “That’s the beginning.”

Photo of Santiago student protesters walking as police fire water canons and tear gas fills the air, Aug. 8, 2012 by AP/Luis Hidalgo.

April 19 2012

13:34

When data goes bad

Data is so central to the decision-making that shapes our countries, jobs and even personal lives that an increasing amount of data journalism involves scrutinising the problems with the very data itself. Here’s an illustrative list of when bad data becomes the story – and the lessons they can teach data journalists:

Deaths in police custody unrecorded

This investigation by the Bureau of Investigative Journalism demonstrates an important question to ask about data: who decides what gets recorded?

In this case, the BIJ identified “a number of cases not included in the official tally of 16 ‘restraint-related’ deaths in the decade to 2009 … Some cases were not included because the person has not been officially arrested or detained.”

As they explain:

“It turns out the IPCC has a very tight definition of ‘in custody’ –  defined only as when someone has been formally arrested or detained under the mental health act. This does not include people who have died after being in contact with the police.

“There are in fact two lists. The one which includes the widely quoted list of sixteen deaths in custody only records the cases where the person has been arrested or detained under the mental health act. So, an individual who comes into contact with the police – is never arrested or detained – but nonetheless dies after being restrained, is not included in the figures.

“… But even using the IPCC’s tightly drawn definition, the Bureau has identified cases that are still missing.”

Cross-checking the official statistics against wider reports was key technique. As was using the Freedom of Information Act to request the details behind them and the details of those “ who died in circumstances where restraint was used but was not necessarily a direct cause of death”.

Cooking the books on drug-related murders

Drug related murders in Mexico
Cross-checking statistics against reports was also used in this investigation by Diego Valle-Jones into Mexican drug deaths:

“The Acteal massacre committed by paramilitary units with government backing against 45 Tzotzil Indians is missing from the vital statistics database. According to the INEGI there were only 2 deaths during December 1997 in the municipality of Chenalho, where the massacre occurred. What a silly way to avoid recording homicides! Now it is just a question of which data is less corrupt.”

Diego also used the Benford’s Law technique to identify potentially fraudulent data, which was also used to highlight relationships between dodgy company data and real world events such as the dotcom bubble and deregulation.

Poor records mean no checks

Detective Inspector Philip Shakesheff exposed a “gap between [local authority] records and police data”, reported The Sunday Times in a story headlined ‘Care home loses child 130 times‘:

“The true scale of the problem was revealed after a check of records on police computers. For every child officially recorded by local authorities as missing in 2010, another seven were unaccounted for without their absence being noted.”

Why is it important?

“The number who go missing is one of the indicators on which Ofsted judges how well children’s homes are performing and the homes have a legal duty to keep accurate records.

“However, there is evidence some homes are failing to do so. In one case, Ofsted gave a good report to a private children’s home in Worcestershire when police records showed 1,630 missing person reports in five years. Police stationed an officer at the home and pressed Ofsted to look closer. The home was downgraded to inadequate and it later closed.

“The risks of being missing from care are demonstrated by Zoe Thomsett, 17, who was Westminster council’s responsibility. It sent her to a care home in Herefordshire, where she went missing several times, the final time for three days. She had earlier been found at an address in Hereford, but because no record was kept, nobody checked the address. She died there of a drugs overdose.

“The troubled life of Dane Edgar, 14, ended with a drugs overdose at a friend’s house after he repeatedly went missing from a children’s home in Northumberland. Another 14-year-old, James Jordan, was killed when he absconded from care and was the passenger in a stolen car.”

Interests not registered

When there are no formal checks on declarations of interest, how can we rely on it? In Chile, the Ciudadano Inteligente Fundaciondecided to check the Chilean MPs’ register of assets and interests by building a database:

“No-one was analysing this data, so it was incomplete,” explained Felipe Heusser, executive president of the Fundacion. “We used technology to build a database, using a wide range of open data and mapped all the MPs’ interests. From that, we found that nearly 40% of MPs were not disclosing their assets fully.”

The organisation has now launched a database that “enables members of the public to find potential conflicts of interest by analysing the data disclosed through the members’ register of assets.”

Data laundering

Tony Hirst’s post about how dodgy data was “laundered” by Facebook in a consultants report is a good illustration of the need to ‘follow the data’.

We have some dodgy evidence, about which we’re biased, so we give it to an “independent” consultant who re-reports it, albeit with caveats, that we can then report, minus the caveats. Lovely, clean evidence. Our lobbyists can then go to a lazy policy researcher and take this scrubbed evidence, referencing it as finding in the Deloitte report, so that it can make its way into a policy briefing.”

“Things just don’t add up”

In the video below Ellen Miller of the Sunlight Foundation takes the US government to task over the inconsistencies in its transparency agenda, and the flawed data published on its USAspending.gov – so flawed that they launched the Clearspending website to automate and highlight the discrepancy between two sources of the same data:

Key budget decisions made on useless data

Sometimes data might appear to tell an astonishing story, but this turns out to be a mistake – and that mistake itself leads you to something much more newsworthy, as Channel 4′s FactCheck found when it started trying to find out if councils had been cutting spending on Sure Start children’s centres:

“That ought to be fairly straightforward, as all councils by law have to fill in something called a Section 251 workbook detailing how much they are spending on various services for young people.

“… Brent Council in north London appeared to have slashed its funding by nearly 90 per cent, something that seemed strange, as we hadn’t heard an outcry from local parents.

“The council swiftly admitted making an accounting error – to the tune of a staggering £6m.”

And they weren’t the only ones. In fact, the Department for Education  admitted the numbers were “not very accurate”:

“So to recap, these spending figures don’t actually reflect the real amount of money spent; figures from different councils are not comparable with each other; spending in one year can’t be compared usefully with other years; and the government doesn’t propose to audit the figures or correct them when they’re wrong.”

This was particularly important because the S251 form “is the document the government uses to reallocate funding from council-run schools to its flagship academies.”:

“The Local Government Association (LGA) says less than £250m should be swiped from council budgets and given to academies, while the government wants to cut more than £1bn, prompting accusations that it is overfunding its favoured schools to the detriment of thousands of other children.

“Many councils’ complaints, made plain in responses to an ongoing government consultation, hinge on DfE’s use of S251, a document it has variously described as “unaudited”, “flawed” and”not fit for purpose”.

No data is still a story

Sticking with education, the TES reports on the outcome of an FOI request on the experience of Ofsted inspectors:

“[Stephen] Ball submitted a Freedom of Information request, asking how many HMIs had experience of being a secondary head, and how many of those had led an outstanding school. The answer? Ofsted “does not hold the details”.

““Secondary heads and academy principals need to be reassured that their work is judged by people who understand its complexity,” Mr Ball said. “Training as a good head of department or a primary school leader on the framework is no longer adequate. Secondary heads don’t fear judgement, but they expect to be judged by people who have experience as well as a theoretical training. After all, a working knowledge of the highway code doesn’t qualify you to become a driving examiner.”

“… Sir Michael Wilshaw, Ofsted’s new chief inspector, has already argued publicly that raw data are a key factor in assessing a school’s performance. By not providing the facts to back up its boasts about the expertise of its inspectors, many heads will remain sceptical of the watchdog’s claims.”

Men aren’t as tall as they say they are

To round off, here’s a quirky piece of data journalism by dating site OkCupid, which looked at the height of its members and found an interesting pattern:

Male height distribution on OKCupid

 

“The male heights on OkCupid very nearly follow the expected normal distribution—except the whole thing is shifted to the right of where it should be.

“Almost universally guys like to add a couple inches. You can also see a more subtle vanity at work: starting at roughly 5′ 8″, the top of the dotted curve tilts even further rightward. This means that guys as they get closer to six feet round up a bit more than usual, stretching for that coveted psychological benchmark.”

Do you know of any other examples of bad data forming the basis of a story? Please post a comment – I’m collecting examples.

13:34

When data goes bad

Data is so central to the decision-making that shapes our countries, jobs and even personal lives that an increasing amount of data journalism involves scrutinising the problems with the very data itself. Here’s an illustrative list of when bad data becomes the story – and the lessons they can teach data journalists:

Deaths in police custody unrecorded

This investigation by the Bureau of Investigative Journalism demonstrates an important question to ask about data: who decides what gets recorded?

In this case, the BIJ identified “a number of cases not included in the official tally of 16 ‘restraint-related’ deaths in the decade to 2009 … Some cases were not included because the person has not been officially arrested or detained.”

As they explain:

“It turns out the IPCC has a very tight definition of ‘in custody’ –  defined only as when someone has been formally arrested or detained under the mental health act. This does not include people who have died after being in contact with the police.

“There are in fact two lists. The one which includes the widely quoted list of sixteen deaths in custody only records the cases where the person has been arrested or detained under the mental health act. So, an individual who comes into contact with the police – is never arrested or detained – but nonetheless dies after being restrained, is not included in the figures.

“… But even using the IPCC’s tightly drawn definition, the Bureau has identified cases that are still missing.”

Cross-checking the official statistics against wider reports was key technique. As was using the Freedom of Information Act to request the details behind them and the details of those “ who died in circumstances where restraint was used but was not necessarily a direct cause of death”.

Cooking the books on drug-related murders

Drug related murders in Mexico
Cross-checking statistics against reports was also used in this investigation by Diego Valle-Jones into Mexican drug deaths:

“The Acteal massacre committed by paramilitary units with government backing against 45 Tzotzil Indians is missing from the vital statistics database. According to the INEGI there were only 2 deaths during December 1997 in the municipality of Chenalho, where the massacre occurred. What a silly way to avoid recording homicides! Now it is just a question of which data is less corrupt.”

Diego also used the Benford’s Law technique to identify potentially fraudulent data, which was also used to highlight relationships between dodgy company data and real world events such as the dotcom bubble and deregulation.

Poor records mean no checks

Detective Inspector Philip Shakesheff exposed a “gap between [local authority] records and police data”, reported The Sunday Times in a story headlined ‘Care home loses child 130 times‘:

“The true scale of the problem was revealed after a check of records on police computers. For every child officially recorded by local authorities as missing in 2010, another seven were unaccounted for without their absence being noted.”

Why is it important?

“The number who go missing is one of the indicators on which Ofsted judges how well children’s homes are performing and the homes have a legal duty to keep accurate records.

“However, there is evidence some homes are failing to do so. In one case, Ofsted gave a good report to a private children’s home in Worcestershire when police records showed 1,630 missing person reports in five years. Police stationed an officer at the home and pressed Ofsted to look closer. The home was downgraded to inadequate and it later closed.

“The risks of being missing from care are demonstrated by Zoe Thomsett, 17, who was Westminster council’s responsibility. It sent her to a care home in Herefordshire, where she went missing several times, the final time for three days. She had earlier been found at an address in Hereford, but because no record was kept, nobody checked the address. She died there of a drugs overdose.

“The troubled life of Dane Edgar, 14, ended with a drugs overdose at a friend’s house after he repeatedly went missing from a children’s home in Northumberland. Another 14-year-old, James Jordan, was killed when he absconded from care and was the passenger in a stolen car.”

Interests not registered

When there are no formal checks on declarations of interest, how can we rely on it? In Chile, the Ciudadano Inteligente Fundaciondecided to check the Chilean MPs’ register of assets and interests by building a database:

“No-one was analysing this data, so it was incomplete,” explained Felipe Heusser, executive president of the Fundacion. “We used technology to build a database, using a wide range of open data and mapped all the MPs’ interests. From that, we found that nearly 40% of MPs were not disclosing their assets fully.”

The organisation has now launched a database that “enables members of the public to find potential conflicts of interest by analysing the data disclosed through the members’ register of assets.”

Data laundering

Tony Hirst’s post about how dodgy data was “laundered” by Facebook in a consultants report is a good illustration of the need to ‘follow the data’.

We have some dodgy evidence, about which we’re biased, so we give it to an “independent” consultant who re-reports it, albeit with caveats, that we can then report, minus the caveats. Lovely, clean evidence. Our lobbyists can then go to a lazy policy researcher and take this scrubbed evidence, referencing it as finding in the Deloitte report, so that it can make its way into a policy briefing.”

“Things just don’t add up”

In the video below Ellen Miller of the Sunlight Foundation takes the US government to task over the inconsistencies in its transparency agenda, and the flawed data published on its USAspending.gov – so flawed that they launched the Clearspending website to automate and highlight the discrepancy between two sources of the same data:

Key budget decisions made on useless data

Sometimes data might appear to tell an astonishing story, but this turns out to be a mistake – and that mistake itself leads you to something much more newsworthy, as Channel 4′s FactCheck found when it started trying to find out if councils had been cutting spending on Sure Start children’s centres:

“That ought to be fairly straightforward, as all councils by law have to fill in something called a Section 251 workbook detailing how much they are spending on various services for young people.

“… Brent Council in north London appeared to have slashed its funding by nearly 90 per cent, something that seemed strange, as we hadn’t heard an outcry from local parents.

“The council swiftly admitted making an accounting error – to the tune of a staggering £6m.”

And they weren’t the only ones. In fact, the Department for Education  admitted the numbers were “not very accurate”:

“So to recap, these spending figures don’t actually reflect the real amount of money spent; figures from different councils are not comparable with each other; spending in one year can’t be compared usefully with other years; and the government doesn’t propose to audit the figures or correct them when they’re wrong.”

This was particularly important because the S251 form “is the document the government uses to reallocate funding from council-run schools to its flagship academies.”:

“The Local Government Association (LGA) says less than £250m should be swiped from council budgets and given to academies, while the government wants to cut more than £1bn, prompting accusations that it is overfunding its favoured schools to the detriment of thousands of other children.

“Many councils’ complaints, made plain in responses to an ongoing government consultation, hinge on DfE’s use of S251, a document it has variously described as “unaudited”, “flawed” and”not fit for purpose”.

No data is still a story

Sticking with education, the TES reports on the outcome of an FOI request on the experience of Ofsted inspectors:

“[Stephen] Ball submitted a Freedom of Information request, asking how many HMIs had experience of being a secondary head, and how many of those had led an outstanding school. The answer? Ofsted “does not hold the details”.

““Secondary heads and academy principals need to be reassured that their work is judged by people who understand its complexity,” Mr Ball said. “Training as a good head of department or a primary school leader on the framework is no longer adequate. Secondary heads don’t fear judgement, but they expect to be judged by people who have experience as well as a theoretical training. After all, a working knowledge of the highway code doesn’t qualify you to become a driving examiner.”

“… Sir Michael Wilshaw, Ofsted’s new chief inspector, has already argued publicly that raw data are a key factor in assessing a school’s performance. By not providing the facts to back up its boasts about the expertise of its inspectors, many heads will remain sceptical of the watchdog’s claims.”

Men aren’t as tall as they say they are

To round off, here’s a quirky piece of data journalism by dating site OkCupid, which looked at the height of its members and found an interesting pattern:

Male height distribution on OKCupid

 

“The male heights on OkCupid very nearly follow the expected normal distribution—except the whole thing is shifted to the right of where it should be.

“Almost universally guys like to add a couple inches. You can also see a more subtle vanity at work: starting at roughly 5′ 8″, the top of the dotted curve tilts even further rightward. This means that guys as they get closer to six feet round up a bit more than usual, stretching for that coveted psychological benchmark.”

Do you know of any other examples of bad data forming the basis of a story? Please post a comment – I’m collecting examples.

09:08

When data goes bad

Incorrect-statistics

Image by Lauren York

Data is so central to the decision-making that shapes our countries, jobs and even personal lives that an increasing amount of data journalism involves scrutinising the problems with the very data itself. Here’s an illustrative list of when bad data becomes the story – and the lessons they can teach data journalists:

Deaths in police custody unrecorded

This investigation by the Bureau of Investigative Journalism demonstrates an important question to ask about data: who decides what gets recorded?

In this case, the BIJ identified “a number of cases not included in the official tally of 16 ‘restraint-related’ deaths in the decade to 2009 … Some cases were not included because the person has not been officially arrested or detained.”

As they explain:

“It turns out the IPCC has a very tight definition of ‘in custody’ –  defined only as when someone has been formally arrested or detained under the mental health act. This does not include people who have died after being in contact with the police.

“There are in fact two lists. The one which includes the widely quoted list of sixteen deaths in custody only records the cases where the person has been arrested or detained under the mental health act. So, an individual who comes into contact with the police – is never arrested or detained – but nonetheless dies after being restrained, is not included in the figures.

“… But even using the IPCC’s tightly drawn definition, the Bureau has identified cases that are still missing.”

Cross-checking the official statistics against wider reports was key technique. As was using the Freedom of Information Act to request the details behind them and the details of those “ who died in circumstances where restraint was used but was not necessarily a direct cause of death”.

Cooking the books on drug-related murders

Drug related murders in Mexico
Cross-checking statistics against reports was also used in this investigation by Diego Valle-Jones into Mexican drug deaths:

“The Acteal massacre committed by paramilitary units with government backing against 45 Tzotzil Indians is missing from the vital statistics database. According to the INEGI there were only 2 deaths during December 1997 in the municipality of Chenalho, where the massacre occurred. What a silly way to avoid recording homicides! Now it is just a question of which data is less corrupt.”

Diego also used the Benford’s Law technique to identify potentially fraudulent data, which was also used to highlight relationships between dodgy company data and real world events such as the dotcom bubble and deregulation.

Poor records mean no checks

Detective Inspector Philip Shakesheff exposed a “gap between [local authority] records and police data”, reported The Sunday Times in a story headlined ‘Care home loses child 130 times‘:

“The true scale of the problem was revealed after a check of records on police computers. For every child officially recorded by local authorities as missing in 2010, another seven were unaccounted for without their absence being noted.”

Why is it important?

“The number who go missing is one of the indicators on which Ofsted judges how well children’s homes are performing and the homes have a legal duty to keep accurate records.

“However, there is evidence some homes are failing to do so. In one case, Ofsted gave a good report to a private children’s home in Worcestershire when police records showed 1,630 missing person reports in five years. Police stationed an officer at the home and pressed Ofsted to look closer. The home was downgraded to inadequate and it later closed.

“The risks of being missing from care are demonstrated by Zoe Thomsett, 17, who was Westminster council’s responsibility. It sent her to a care home in Herefordshire, where she went missing several times, the final time for three days. She had earlier been found at an address in Hereford, but because no record was kept, nobody checked the address. She died there of a drugs overdose.

“The troubled life of Dane Edgar, 14, ended with a drugs overdose at a friend’s house after he repeatedly went missing from a children’s home in Northumberland. Another 14-year-old, James Jordan, was killed when he absconded from care and was the passenger in a stolen car.”

Interests not registered

When there are no formal checks on declarations of interest, how can we rely on it? In Chile, the Ciudadano Inteligente Fundaciondecided to check the Chilean MPs’ register of assets and interests by building a database:

“No-one was analysing this data, so it was incomplete,” explained Felipe Heusser, executive president of the Fundacion. “We used technology to build a database, using a wide range of open data and mapped all the MPs’ interests. From that, we found that nearly 40% of MPs were not disclosing their assets fully.”

The organisation has now launched a database that “enables members of the public to find potential conflicts of interest by analysing the data disclosed through the members’ register of assets.”

Data laundering

Tony Hirst’s post about how dodgy data was “laundered” by Facebook in a consultants report is a good illustration of the need to ‘follow the data’.

We have some dodgy evidence, about which we’re biased, so we give it to an “independent” consultant who re-reports it, albeit with caveats, that we can then report, minus the caveats. Lovely, clean evidence. Our lobbyists can then go to a lazy policy researcher and take this scrubbed evidence, referencing it as finding in the Deloitte report, so that it can make its way into a policy briefing.”

“Things just don’t add up”

In the video below Ellen Miller of the Sunlight Foundation takes the US government to task over the inconsistencies in its transparency agenda, and the flawed data published on its USAspending.gov – so flawed that they launched the Clearspending website to automate and highlight the discrepancy between two sources of the same data:

Key budget decisions made on useless data

Sometimes data might appear to tell an astonishing story, but this turns out to be a mistake – and that mistake itself leads you to something much more newsworthy, as Channel 4′s FactCheck foundwhen it started trying to find out if councils had been cutting spending on Sure Start children’s centres:

“That ought to be fairly straightforward, as all councils by law have to fill in something called a Section 251 workbook detailing how much they are spending on various services for young people.

“… Brent Council in north London appeared to have slashed its funding by nearly 90 per cent, something that seemed strange, as we hadn’t heard an outcry from local parents.

“The council swiftly admitted making an accounting error – to the tune of a staggering £6m.”

And they weren’t the only ones. In fact, the Department for Education  admitted the numbers were “not very accurate”:

“So to recap, these spending figures don’t actually reflect the real amount of money spent; figures from different councils are not comparable with each other; spending in one year can’t be compared usefully with other years; and the government doesn’t propose to audit the figures or correct them when they’re wrong.”

This was particularly important because the S251 form “is the document the government uses to reallocate funding from council-run schools to its flagship academies.”:

“The Local Government Association (LGA) says less than £250m should be swiped from council budgets and given to academies, while the government wants to cut more than £1bn, prompting accusations that it is overfunding its favoured schools to the detriment of thousands of other children.

“Many councils’ complaints, made plain in responses to an ongoing government consultation, hinge on DfE’s use of S251, a document it has variously described as “unaudited”, “flawed” and”not fit for purpose”.

No data is still a story

Sticking with education, the TES reports on the outcome of an FOI request on the experience of Ofsted inspectors:

“[Stephen] Ball submitted a Freedom of Information request, asking how many HMIs had experience of being a secondary head, and how many of those had led an outstanding school. The answer? Ofsted “does not hold the details”.

““Secondary heads and academy principals need to be reassured that their work is judged by people who understand its complexity,” Mr Ball said. “Training as a good head of department or a primary school leader on the framework is no longer adequate. Secondary heads don’t fear judgement, but they expect to be judged by people who have experience as well as a theoretical training. After all, a working knowledge of the highway code doesn’t qualify you to become a driving examiner.”

“… Sir Michael Wilshaw, Ofsted’s new chief inspector, has already argued publicly that raw data are a key factor in assessing a school’s performance. By not providing the facts to back up its boasts about the expertise of its inspectors, many heads will remain sceptical of the watchdog’s claims.”

Men aren’t as tall as they say they are

To round off, here’s a quirky piece of data journalism by dating site OkCupid, which looked at the height of its members and found an interesting pattern:

Male height distribution on OKCupid

“The male heights on OkCupid very nearly follow the expected normal distribution—except the whole thing is shifted to the right of where it should be.

“Almost universally guys like to add a couple inches. You can also see a more subtle vanity at work: starting at roughly 5′ 8″, the top of the dotted curve tilts even further rightward. This means that guys as they get closer to six feet round up a bit more than usual, stretching for that coveted psychological benchmark.”

Do you know of any other examples of bad data forming the basis of a story? Please post a comment – I’m collecting examples.

UPDATE (April 20 2012): A useful addition from Simon Rogers: Named and shamed: the worst government annual reports explains why government department spending reports fail to support the Government’s claimed desire for an “army of armchair auditors”, with a list of the worst offenders at the end.

Also:

09:08

When data goes bad

Incorrect-statistics

Image by Lauren York

Data is so central to the decision-making that shapes our countries, jobs and even personal lives that an increasing amount of data journalism involves scrutinising the problems with the very data itself. Here’s an illustrative list of when bad data becomes the story – and the lessons they can teach data journalists:

Deaths in police custody unrecorded

This investigation by the Bureau of Investigative Journalism demonstrates an important question to ask about data: who decides what gets recorded?

In this case, the BIJ identified “a number of cases not included in the official tally of 16 ‘restraint-related’ deaths in the decade to 2009 … Some cases were not included because the person has not been officially arrested or detained.”

As they explain:

“It turns out the IPCC has a very tight definition of ‘in custody’ –  defined only as when someone has been formally arrested or detained under the mental health act. This does not include people who have died after being in contact with the police.

“There are in fact two lists. The one which includes the widely quoted list of sixteen deaths in custody only records the cases where the person has been arrested or detained under the mental health act. So, an individual who comes into contact with the police – is never arrested or detained – but nonetheless dies after being restrained, is not included in the figures.

“… But even using the IPCC’s tightly drawn definition, the Bureau has identified cases that are still missing.”

Cross-checking the official statistics against wider reports was key technique. As was using the Freedom of Information Act to request the details behind them and the details of those “ who died in circumstances where restraint was used but was not necessarily a direct cause of death”.

Cooking the books on drug-related murders

Drug related murders in Mexico
Cross-checking statistics against reports was also used in this investigation by Diego Valle-Jones into Mexican drug deaths:

“The Acteal massacre committed by paramilitary units with government backing against 45 Tzotzil Indians is missing from the vital statistics database. According to the INEGI there were only 2 deaths during December 1997 in the municipality of Chenalho, where the massacre occurred. What a silly way to avoid recording homicides! Now it is just a question of which data is less corrupt.”

Diego also used the Benford’s Law technique to identify potentially fraudulent data, which was also used to highlight relationships between dodgy company data and real world events such as the dotcom bubble and deregulation.

Poor records mean no checks

Detective Inspector Philip Shakesheff exposed a “gap between [local authority] records and police data”, reported The Sunday Times in a story headlined ‘Care home loses child 130 times‘:

“The true scale of the problem was revealed after a check of records on police computers. For every child officially recorded by local authorities as missing in 2010, another seven were unaccounted for without their absence being noted.”

Why is it important?

“The number who go missing is one of the indicators on which Ofsted judges how well children’s homes are performing and the homes have a legal duty to keep accurate records.

“However, there is evidence some homes are failing to do so. In one case, Ofsted gave a good report to a private children’s home in Worcestershire when police records showed 1,630 missing person reports in five years. Police stationed an officer at the home and pressed Ofsted to look closer. The home was downgraded to inadequate and it later closed.

“The risks of being missing from care are demonstrated by Zoe Thomsett, 17, who was Westminster council’s responsibility. It sent her to a care home in Herefordshire, where she went missing several times, the final time for three days. She had earlier been found at an address in Hereford, but because no record was kept, nobody checked the address. She died there of a drugs overdose.

“The troubled life of Dane Edgar, 14, ended with a drugs overdose at a friend’s house after he repeatedly went missing from a children’s home in Northumberland. Another 14-year-old, James Jordan, was killed when he absconded from care and was the passenger in a stolen car.”

Interests not registered

When there are no formal checks on declarations of interest, how can we rely on it? In Chile, the Ciudadano Inteligente Fundaciondecided to check the Chilean MPs’ register of assets and interests by building a database:

“No-one was analysing this data, so it was incomplete,” explained Felipe Heusser, executive president of the Fundacion. “We used technology to build a database, using a wide range of open data and mapped all the MPs’ interests. From that, we found that nearly 40% of MPs were not disclosing their assets fully.”

The organisation has now launched a database that “enables members of the public to find potential conflicts of interest by analysing the data disclosed through the members’ register of assets.”

Data laundering

Tony Hirst’s post about how dodgy data was “laundered” by Facebook in a consultants report is a good illustration of the need to ‘follow the data’.

We have some dodgy evidence, about which we’re biased, so we give it to an “independent” consultant who re-reports it, albeit with caveats, that we can then report, minus the caveats. Lovely, clean evidence. Our lobbyists can then go to a lazy policy researcher and take this scrubbed evidence, referencing it as finding in the Deloitte report, so that it can make its way into a policy briefing.”

“Things just don’t add up”

In the video below Ellen Miller of the Sunlight Foundation takes the US government to task over the inconsistencies in its transparency agenda, and the flawed data published on its USAspending.gov – so flawed that they launched the Clearspending website to automate and highlight the discrepancy between two sources of the same data:

Key budget decisions made on useless data

Sometimes data might appear to tell an astonishing story, but this turns out to be a mistake – and that mistake itself leads you to something much more newsworthy, as Channel 4′s FactCheck foundwhen it started trying to find out if councils had been cutting spending on Sure Start children’s centres:

“That ought to be fairly straightforward, as all councils by law have to fill in something called a Section 251 workbook detailing how much they are spending on various services for young people.

“… Brent Council in north London appeared to have slashed its funding by nearly 90 per cent, something that seemed strange, as we hadn’t heard an outcry from local parents.

“The council swiftly admitted making an accounting error – to the tune of a staggering £6m.”

And they weren’t the only ones. In fact, the Department for Education  admitted the numbers were “not very accurate”:

“So to recap, these spending figures don’t actually reflect the real amount of money spent; figures from different councils are not comparable with each other; spending in one year can’t be compared usefully with other years; and the government doesn’t propose to audit the figures or correct them when they’re wrong.”

This was particularly important because the S251 form “is the document the government uses to reallocate funding from council-run schools to its flagship academies.”:

“The Local Government Association (LGA) says less than £250m should be swiped from council budgets and given to academies, while the government wants to cut more than £1bn, prompting accusations that it is overfunding its favoured schools to the detriment of thousands of other children.

“Many councils’ complaints, made plain in responses to an ongoing government consultation, hinge on DfE’s use of S251, a document it has variously described as “unaudited”, “flawed” and”not fit for purpose”.

No data is still a story

Sticking with education, the TES reports on the outcome of an FOI request on the experience of Ofsted inspectors:

“[Stephen] Ball submitted a Freedom of Information request, asking how many HMIs had experience of being a secondary head, and how many of those had led an outstanding school. The answer? Ofsted “does not hold the details”.

““Secondary heads and academy principals need to be reassured that their work is judged by people who understand its complexity,” Mr Ball said. “Training as a good head of department or a primary school leader on the framework is no longer adequate. Secondary heads don’t fear judgement, but they expect to be judged by people who have experience as well as a theoretical training. After all, a working knowledge of the highway code doesn’t qualify you to become a driving examiner.”

“… Sir Michael Wilshaw, Ofsted’s new chief inspector, has already argued publicly that raw data are a key factor in assessing a school’s performance. By not providing the facts to back up its boasts about the expertise of its inspectors, many heads will remain sceptical of the watchdog’s claims.”

Men aren’t as tall as they say they are

To round off, here’s a quirky piece of data journalism by dating site OkCupid, which looked at the height of its members and found an interesting pattern:

Male height distribution on OKCupid

“The male heights on OkCupid very nearly follow the expected normal distribution—except the whole thing is shifted to the right of where it should be.

“Almost universally guys like to add a couple inches. You can also see a more subtle vanity at work: starting at roughly 5′ 8″, the top of the dotted curve tilts even further rightward. This means that guys as they get closer to six feet round up a bit more than usual, stretching for that coveted psychological benchmark.”

Do you know of any other examples of bad data forming the basis of a story? Please post a comment – I’m collecting examples.

UPDATE (April 20 2012): A useful addition from Simon Rogers: Named and shamed: the worst government annual reports explains why government department spending reports fail to support the Government’s claimed desire for an “army of armchair auditors”, with a list of the worst offenders at the end.

Also:


Filed under: online journalism Tagged: bad data, benford's law, BIJ, bureau of investigative journalism, Channel 4, Chile, Ciudadano Inteligente Fundacion, Clearspending, data laundering, dating, Deaths in custody, ellen miller, FactCheck, Felipe Heusser, height, IPCC, Lauren York, missing children, OKCupid, Philip Shakesheff, register of interests, S251, sex trafficking, simon rogers, sunday times, sunlight foundation, tony hirst
09:08

When data goes bad

Incorrect-statistics

Image by Lauren York

Data is so central to the decision-making that shapes our countries, jobs and even personal lives that an increasing amount of data journalism involves scrutinising the problems with the very data itself. Here’s an illustrative list of when bad data becomes the story – and the lessons they can teach data journalists:

Deaths in police custody unrecorded

This investigation by the Bureau of Investigative Journalism demonstrates an important question to ask about data: who decides what gets recorded?

In this case, the BIJ identified “a number of cases not included in the official tally of 16 ‘restraint-related’ deaths in the decade to 2009 … Some cases were not included because the person has not been officially arrested or detained.”

As they explain:

“It turns out the IPCC has a very tight definition of ‘in custody’ –  defined only as when someone has been formally arrested or detained under the mental health act. This does not include people who have died after being in contact with the police.

“There are in fact two lists. The one which includes the widely quoted list of sixteen deaths in custody only records the cases where the person has been arrested or detained under the mental health act. So, an individual who comes into contact with the police – is never arrested or detained – but nonetheless dies after being restrained, is not included in the figures.

“… But even using the IPCC’s tightly drawn definition, the Bureau has identified cases that are still missing.”

Cross-checking the official statistics against wider reports was key technique. As was using the Freedom of Information Act to request the details behind them and the details of those “ who died in circumstances where restraint was used but was not necessarily a direct cause of death”.

Cooking the books on drug-related murders

Drug related murders in Mexico
Cross-checking statistics against reports was also used in this investigation by Diego Valle-Jones into Mexican drug deaths:

“The Acteal massacre committed by paramilitary units with government backing against 45 Tzotzil Indians is missing from the vital statistics database. According to the INEGI there were only 2 deaths during December 1997 in the municipality of Chenalho, where the massacre occurred. What a silly way to avoid recording homicides! Now it is just a question of which data is less corrupt.”

Diego also used the Benford’s Law technique to identify potentially fraudulent data, which was also used to highlight relationships between dodgy company data and real world events such as the dotcom bubble and deregulation.

Poor records mean no checks

Detective Inspector Philip Shakesheff exposed a “gap between [local authority] records and police data”, reported The Sunday Times in a story headlined ‘Care home loses child 130 times‘:

“The true scale of the problem was revealed after a check of records on police computers. For every child officially recorded by local authorities as missing in 2010, another seven were unaccounted for without their absence being noted.”

Why is it important?

“The number who go missing is one of the indicators on which Ofsted judges how well children’s homes are performing and the homes have a legal duty to keep accurate records.

“However, there is evidence some homes are failing to do so. In one case, Ofsted gave a good report to a private children’s home in Worcestershire when police records showed 1,630 missing person reports in five years. Police stationed an officer at the home and pressed Ofsted to look closer. The home was downgraded to inadequate and it later closed.

“The risks of being missing from care are demonstrated by Zoe Thomsett, 17, who was Westminster council’s responsibility. It sent her to a care home in Herefordshire, where she went missing several times, the final time for three days. She had earlier been found at an address in Hereford, but because no record was kept, nobody checked the address. She died there of a drugs overdose.

“The troubled life of Dane Edgar, 14, ended with a drugs overdose at a friend’s house after he repeatedly went missing from a children’s home in Northumberland. Another 14-year-old, James Jordan, was killed when he absconded from care and was the passenger in a stolen car.”

Interests not registered

When there are no formal checks on declarations of interest, how can we rely on it? In Chile, the Ciudadano Inteligente Fundaciondecided to check the Chilean MPs’ register of assets and interests by building a database:

“No-one was analysing this data, so it was incomplete,” explained Felipe Heusser, executive president of the Fundacion. “We used technology to build a database, using a wide range of open data and mapped all the MPs’ interests. From that, we found that nearly 40% of MPs were not disclosing their assets fully.”

The organisation has now launched a database that “enables members of the public to find potential conflicts of interest by analysing the data disclosed through the members’ register of assets.”

Data laundering

Tony Hirst’s post about how dodgy data was “laundered” by Facebook in a consultants report is a good illustration of the need to ‘follow the data’.

We have some dodgy evidence, about which we’re biased, so we give it to an “independent” consultant who re-reports it, albeit with caveats, that we can then report, minus the caveats. Lovely, clean evidence. Our lobbyists can then go to a lazy policy researcher and take this scrubbed evidence, referencing it as finding in the Deloitte report, so that it can make its way into a policy briefing.”

“Things just don’t add up”

In the video below Ellen Miller of the Sunlight Foundation takes the US government to task over the inconsistencies in its transparency agenda, and the flawed data published on its USAspending.gov – so flawed that they launched the Clearspending website to automate and highlight the discrepancy between two sources of the same data:

Key budget decisions made on useless data

Sometimes data might appear to tell an astonishing story, but this turns out to be a mistake – and that mistake itself leads you to something much more newsworthy, as Channel 4′s FactCheck foundwhen it started trying to find out if councils had been cutting spending on Sure Start children’s centres:

“That ought to be fairly straightforward, as all councils by law have to fill in something called a Section 251 workbook detailing how much they are spending on various services for young people.

“… Brent Council in north London appeared to have slashed its funding by nearly 90 per cent, something that seemed strange, as we hadn’t heard an outcry from local parents.

“The council swiftly admitted making an accounting error – to the tune of a staggering £6m.”

And they weren’t the only ones. In fact, the Department for Education  admitted the numbers were “not very accurate”:

“So to recap, these spending figures don’t actually reflect the real amount of money spent; figures from different councils are not comparable with each other; spending in one year can’t be compared usefully with other years; and the government doesn’t propose to audit the figures or correct them when they’re wrong.”

This was particularly important because the S251 form “is the document the government uses to reallocate funding from council-run schools to its flagship academies.”:

“The Local Government Association (LGA) says less than £250m should be swiped from council budgets and given to academies, while the government wants to cut more than £1bn, prompting accusations that it is overfunding its favoured schools to the detriment of thousands of other children.

“Many councils’ complaints, made plain in responses to an ongoing government consultation, hinge on DfE’s use of S251, a document it has variously described as “unaudited”, “flawed” and”not fit for purpose”.

No data is still a story

Sticking with education, the TES reports on the outcome of an FOI request on the experience of Ofsted inspectors:

“[Stephen] Ball submitted a Freedom of Information request, asking how many HMIs had experience of being a secondary head, and how many of those had led an outstanding school. The answer? Ofsted “does not hold the details”.

““Secondary heads and academy principals need to be reassured that their work is judged by people who understand its complexity,” Mr Ball said. “Training as a good head of department or a primary school leader on the framework is no longer adequate. Secondary heads don’t fear judgement, but they expect to be judged by people who have experience as well as a theoretical training. After all, a working knowledge of the highway code doesn’t qualify you to become a driving examiner.”

“… Sir Michael Wilshaw, Ofsted’s new chief inspector, has already argued publicly that raw data are a key factor in assessing a school’s performance. By not providing the facts to back up its boasts about the expertise of its inspectors, many heads will remain sceptical of the watchdog’s claims.”

Men aren’t as tall as they say they are

To round off, here’s a quirky piece of data journalism by dating site OkCupid, which looked at the height of its members and found an interesting pattern:

Male height distribution on OKCupid

“The male heights on OkCupid very nearly follow the expected normal distribution—except the whole thing is shifted to the right of where it should be.

“Almost universally guys like to add a couple inches. You can also see a more subtle vanity at work: starting at roughly 5′ 8″, the top of the dotted curve tilts even further rightward. This means that guys as they get closer to six feet round up a bit more than usual, stretching for that coveted psychological benchmark.”

Do you know of any other examples of bad data forming the basis of a story? Please post a comment – I’m collecting examples.

UPDATE (April 20 2012): A useful addition from Simon Rogers: Named and shamed: the worst government annual reports explains why government department spending reports fail to support the Government’s claimed desire for an “army of armchair auditors”, with a list of the worst offenders at the end.

Also:


Filed under: online journalism Tagged: bad data, benford's law, BIJ, bureau of investigative journalism, Channel 4, Chile, Ciudadano Inteligente Fundacion, Clearspending, data laundering, dating, Deaths in custody, ellen miller, FactCheck, Felipe Heusser, height, IPCC, Lauren York, missing children, OKCupid, Philip Shakesheff, register of interests, S251, sex trafficking, simon rogers, sunday times, sunlight foundation, tony hirst

October 09 2011

17:07

How mobile phones could bring public services to people in developing countries

PBS MediaShift :: In Santiago, Chile, more than 60% of the poorest citizens don't have access to the Internet. In the rest of the country, that number increases to 80%, and in rural areas, an Internet connection is almost nonexistent. But there are more than 20 million mobile phones in the nation,that's actually around 1.15 cell phones per capita in a nation of 17,094,270 people. And in rural areas, cell phones are king.

Miguel Paz: What if governments of underdeveloped countries create and provide easy ways to access public information and services on mobile phones with an application or open-source web app that could be downloaded from government websites?

Miguel Paz, Continue to read www.pbs.org

October 06 2011

12:20

How Mobile Phones Could Bring Public Services to People in Developing Countries

In Santiago, Chile, more than 60 percent of the poorest citizens don't have access to the Internet. In the rest of the country, that number increases to 80 percent, and in rural areas, an Internet connection is almost nonexistent. But there are more than 20 million mobile phones in the nation, according to the latest survey by the Undersecretary of Telecommunications. (That's actually around 1.15 cell phones per capita in a nation of 17,094,270 people.) And in rural areas, cell phones are king.

santiago.jpg

As Knight News Challenge winners FrontlineSMS, Ushahidi and NextDrop have shown, mobile communications are crucial for citizens living in rural areas, where being able to reach other people and access relevant news and public services information make a huge improvement in people's lives. Plus, cell phones are tools that most already have.

THE PITCH

What if, apart from efforts to widen connectivity in isolated areas and government programs to provide computers for schools in rural areas (which has been a very good, but slow, undertaking, and not an attractive business for telecom companies), governments of underdeveloped countries create and provide easy ways to access public information and services on mobile phones with an application or open-source web app that could be downloaded from government websites (in Chile it's Gob.cl)? Or cellular service providers could pre-install an app or direct access to a web app on every smartphone or other devices?

This could mean a great deal for people, particularly in rural and impoverished areas where the biggest news is not what's happening in Congress or the presidential palace, but what is happening to you and your community (something Facebook understood very well in its latest change that challenges the notion of what is newsworthy -- but that's a topic for a separate post).

People could do things like schedule a doctor's appointment or receive notice that a doctor won't be available; find out about grants to improve water conditions in their sector; receive direct information about training programs for growing organic food and the market prices for products they might sell; find out how their kids are doing in a school they attend in the city or if the rural bus system will go this week to the nearest town or not. These are just a few very straightforward examples of useful public services information that could be available on people's phones. Such availability of information could save time and money for those who lack both things.

I know it because I saw it as a boy growing up in a small town -- and as the son of a farmer who still hasn't gotten around to the idea of using a computer, despite having the chance to use one. But because my father owns a mobile phone, he's become an expert user of SMS and applications that allow him to check weather conditions.

WHAT'S IN IT FOR THE TELECOM COMPANIES

At the same time, telecom companies could support this initiative by providing mobile Internet connection packages and a free SMS service for rural areas by which citizens could specify their information searches or requests (a kind of help desk). Why would they do it for free? Because with each free transaction, there might be another one that has nothing to do with the government or public services information, which may produce additional income. It might also improve the companies' public image.

Another way of getting support from these companies consists of giving them a
tax reduction for providing the service and automatic updates of information. Thus, rural citizens living in small towns and cities would be able to access the data they need (pension reforms, hospital appointments, housing benefits, food grants, etc).

IN SIMPLE WORDS

To do what we're talking about, we need clean and intuitive interfaces with super-simple steps and strong government websites or apps that learn from the end users' needs, systematizing:

  • Databases containing questions and answers made by ministries and government staff.
  • Services citizens can access in order to ask for all kinds of information: subsidies, hours of service, etc.
  • Simple and complex procedures, so that answers can be delivered accurately and in the shortest amount of time.

This reduces the margin of error, maximizes human resources -- decreasing the man-hours needed for searching for requested information -- allows specific departments to detect questions which are more usual, and meets the needs of users and citizens.

However, in order to make citizens understand the information, it has to be written in a simple way, with no illegible technical or legal terms. For such a purpose, there are citizen language manuals that standardize response criteria issued by the state. (A good example of this in Spanish is the Mexican Lenguaje Ciudadano government guide.)

This is a small civic proposal to start a wider conversation and brainstorming and discover projects and ideas that may already be addressing this issue. Please feel free to post your tips and thoughts in the comments section.

Image of Santiago, Chile by Flickr user Cleanie.

May 11 2011

19:02

No Gloom Here: In Latin America, Newspapers Boom

If you spend much time in U.S. newsrooms these days, you might contract a serious case of gloom and doom. Talk is still focused on declining circulations, aging readerships, and the absence of new business models to pay for the production of quality content.

But it would be a mistake to assume that this is the case for the rest of the world. In fact, in many regions, the newspaper business is booming. Some countries' newspapers are pulling in record advertising and those double-digit profit margins that were common in 1990s America.

I recently had the chance to observe this phenomenon firsthand at the Bogota, Colombia, conference of the World Association of Newspapers and News Publishers (WAN-IFRA), where there was little gloom or doom to be found.

eltiempofrontpagescreengrab.png

Instead, newspapers were reporting extraordinary growth in advertising sales from 2005 to 2009: 62 percent in Argentina, 70 percent in Brazil, and 57 percent in Colombia itself. (These figures, drawn from a ZenithOptimedia forecast, contrasted with 34 percent drops in the U.S. and the U.K. over the same period.)

Newspaper circulation is growing sharply in Brazil (29 percent), modestly in Argentina and Bolivia, and holding steady in Colombia and Chile. (It was down more than 12 percent in the U.S.)

But what's most striking about the Latin American news industry is the sense of dynamism. The digital revolution is coming to Latin America -- but it's arriving hand-in-hand with the news organizations, and that makes all the difference.

Multi-Platform Success

That point was reinforced with a visit to the newsrooms of El Tiempo, Colombia's leading daily. The newspaper understandably prides itself on the way it has implemented newsroom convergence. Its expansive headquarters are a few decades old, but look freshly minted, refitted top to bottom with new technology. They include the daily paper, two television channels (CityTV and Canal El Tiempo), as well as a vast array of online products.

In El Tiempo's model, information is endlessly produced and recirculated across platforms. Pieces that air on the television channels are recut by a team of young online editors into two- and three-minute pieces that can circulate online. Breaking news goes out on Twitter, leading traffic back to the website and the newspaper. Each platform is carefully monitored for editorial quality.

According to newspaper director Roberto Pombo, "We had to appoint a journalist to be our Twitter editor because we had a report that went out on Twitter that diverged from the story on ElTiempo.com. It was a garden-variety error, but it convinced us we needed editors to be responsible for social networks."

Pombo has shaped the paper's news to be platform neutral. "We're going with everything in every medium, and the audience can stay where they are," he said. Pombo said the newspaper El Tiempo, whose staff create much of the core content, generates about a 9 percent profit, which is augmented by profits from the television and online operations. "Our newspaper readers are not diminishing, our online audience is growing, and the ads are holding," he said.

Online earnings are smaller but are growing more rapidly. The company has no plans to charge for online content, but goes to great lengths to leverage cross-promotion.

robertopomboeltiempo.jpg

Spanish Ownership

"You can't carry out convergence as a cost-cutting measure -- but you save money in the long run," Pombo said. "All I care about is that if somebody gets a news update on Twitter and somebody asks, 'Where did you get that,' they answer 'Tiempo.' It's all about the brand."

El Tiempo was founded in 1911 and long operated under the leadership of the Santos family. In 2007 the paper was sold to Planeta, a Spanish publishing group, which had to readjust to the Colombian market.

"The owners are living two realities. There's an economic crisis in Spain, but things are fine here, so we have to explain it to them," Pombo said. Spain's newspapers are suffering worse than those in the U.S.

El Tiempo is not alone in its prosperity. Sebastian Hiller, director of La Vanguardia Liberal in the city of Bucaramanga, said, "Most of the major Colombian papers are making 15-20 percent profits, and some of them 30 percent, especially if they've been investing in convergence." (One exception is the venerable Bogota paper El Espectador, which has recently struggled back from the brink of extinction.)

Slow, Steady Economic Growth Good for News

What explains the robust health of these Latin American news organizations?

The first answer is the local market. The Andean nations have largely dodged the 2008 economic downturn, and have been experiencing steady growth in recent years.

Second, this growth has been more evenly distributed than in the past. Many Latin American countries are seeing incomes rise among the urban poor, and with them disposable income. This is a sweet spot for newspaper sales, since there may be discretionary spending for a daily newspaper, but not enough for a computer and an Internet connection.

In Colombia, as in other Latin American countries, there has been a boom in new tabloids and glossy consumer magazines, many of which subsidize quality broadsheets in the same company. Some of these tabloids have reached circulations of 2 million to 3 million within two years of their launch.

Capturing Digital Sales

Third, and perhaps most intriguing, digital is arriving in Latin America, but more slowly than in the U.S. and Europe. This has allowed news organizations to learn from other markets' mistakes, and claim larger shares of the online advertising space before the search engines and aggregators can dominate it. The managers don't care whether the advertising ends up on paper or online -- as long as it ends up with them.

One of the side benefits of this development is a dramatic rise in quality. A number of papers in the region have expanded their foreign coverage and investigative journalism, and have won the prizes to prove it. (For a striking example, look to Costa Rica's La Nacion, where exemplary reporting in 2004 landed two past presidents in jail.)

This is not to say that everything's rosy south of the border. Mexican newspapers are under attack from narco traffickers and corrupt government officials, while Argentina's leading newspaper, Clarin, is locked in a bitter contest with the government. On the other hand, news media are playing a stronger role in Latin American society than ever before, and their business models may buy them precious time to forge a path into the future.

Anne Nelson is an educator, consultant and author in the field of international media strategy. She created and teaches New Media and Development Communications at Columbia's School of International and Public Affairs (SIPA) and teaches an international teleconference course at Bard College. She is a senior consultant on media, education and philanthropy for Anthony Knerr & Associates. She is on Twitter as @anelsona, was a 2005 Guggenheim Fellow, and is a member of the Council on Foreign Relations.

This is a summary. Visit our site for the full post ».

October 15 2010

10:54

Econsultancy: Criticism of Chilean miners coverage misses the point

Econsultancy’s Patricio Robles responds to criticism of coverage of the Chilean miners’ rescue this week. Some journalism academics called it “a story about journalism’s failure”, but is this negativity part of journalism’s problem, he asks.

While nobody is suggesting that the news media blind itself to the world’s ills and injustices, one should consider that part of the news media’s dilemma is how you sell a product that is often filled to the brim with negative stories – crime, tragedy, political squabbling … The irony, of course, is that you can only sell so much bad news. At some point, people get tired of opening up the newspaper to read about a politician who cheated on his wife and didn’t pay his taxes, or turning on the television and seeing images of “suffering at home.” And let’s not forget about Lindsey Lohan. So what do people do? They cancel their newspaper subscriptions, and they skip past CNN when channel surfing.

Full post on Econsultancy at this link…Similar Posts:



October 14 2010

18:49

THE VIRGINIAN PILOT FRONT PAGE PLAYS GLO-CAL

The story from Chile and the story from the Virginia Hampton’s NASA people helping in Chile.

A great “glo-cal” front page.

Bravo!

(Via Charles Apple)

16:17

‘The journalists had become cameras, not human beings anymore’: reflections on the Chile miners story

The rescue of Chile’s trapped miners captured the attention of the world. Live blogs, 24-hour TV stations, newspapers – the story was embraced by all platforms. Scotland’s Daily Record even featured a picture of the first emerged miner on its 4am front page. The media spotlight was well and truly focused on events at ‘Camp Hope’, enabling people all over the world to witness the remarkable rescue.

But some of the media coverage and in particular the volume of journalists who descended on the mine area has come under fire as questions arise over the necessity of hundreds of reporters being at the location to cover the story.

In an interview with euronews, local journalist Claudia La Torre said the behaviour of journalists desperate to cover the story was “too much”.

Before the media arrived there was a lot of crying, and then the feeling spread and the media got hold of it and put it to the fore. The media has been very important as it has informed everyone. But there are still limits. Yesterday I saw some miner’s families telling the media to go away. They wanted some privacy, the cameras and lights were harassing them. I regretted that, and I felt it was too much. The mother of the first miner rescued shouted at the journalists to stop, she was trying to hold her son in her arms and she couldn’t. I had to walk away, I felt that the journalists had just become cameras and not human beings any more.

Steve Safran from Lost Remote also commented on the amount of coverage and number of journalists at the scene, which he felt were “way out of proportion”.

Not to be cranky here – it’s great that these men are being rescued. But the coverage is way out of proportion to the importance of the event. And there is little perspective here. Suppose these men had died in the collapse back in August. Would it have received a mention at all in the news? This has as much to do with the fact that the coverage could be planned as anything.

Jeremy Littau from Lehigh University added that Chile is a ‘story about journalism’s failure’.

I see a story about journalism. To know that 1300 journalists have descended on this mining town to cover a worldwide story is a little disconcerting in an era of closed foreign bureaus and budget cutbacks. Many might question that thought given the intense interest in the story; my Twitter and Facebook feeds were lit up last night as the first miner descended up the 2000-foot shaft. But the public doesn’t think in terms of resources when it consumes journalism; it only has what it has in front of it.

These concerns continued today as reports that the BBC spent more than £100,000 on covering the rescue operation emerged via a leaked memo from BBC world news editor Jon Williams, which suggested the broadcaster will have to reduce its coverage of other major events as a result.

“The financial situation is serious”, Mr Williams wrote. “We are currently £67k beyond our agreed overspend of £500k; newsgathering’s costs for Chile will exceed £100,000.”

Coverage of the forthcoming Nato summit in Lisbon, the Cancun climate summit and the Davos World Economic Forum will all suffer as a result of the black hole in the corporation’s finances.

But while the rescue operation of the 33 men may be over, the media interest in the miners is likely to continue for some time. In fact, according to a report from the Guardian, freelance journalist Jonathan Franklin, who reported on the story for the newspaper from the start, is already signed up to write a book about events.

“This is one of the great rescue stories of all time,” he said, admitting he himself had wept as the first miners were released on Tuesday night. “It’s the reason we all want to be reporters: a remarkable story of the world coming together for a good reason. It taps into human altruism, the desire to work together, perseverance, faith that good things happen, never giving up.” The early chapters of the book, he said, were already written.

Similar Posts:



October 13 2010

15:14

EUROPEAN NEWSPAPERS FACE THE DAY AFTER CHALLENGE

The rescue news from Chile arrived late to Europe so you will not find any major front page or coverage in today’s editions.

American newspaper had a better chance but in general the pattern was Big Pictures (TV was better) and Big Words (TV was better).

So, what we can expect tomorrow in the front pages of the best European newspapers?

Well, not too much.

They will try again the Big Pictures and Big Words easy game.

Ignoring that the rescue was a worldwide TV event and it’s going to be difficult to add new angles and clues to the big news of the day before…

Yes, I know that this is always difficult, but newspaper editors had many week in advance to plan for this magic moment.

TV did its work.

I watched BBC, News Sky News and CNN Chile and all of them did a superb job.

What European newspapers readers expect tomorrow is not to see again the same pictures, the same infographics, and the same news, but a more creative, analytical, pro-active and “news behind the news” stories.

But I doubt that our newspapers will do it.

Instead, like in music and as our grandmother will tell us when we were little ones, “if you don’t play well, at least play loud”

So expect more empty noise.

March 10 2010

15:03

Help Link People After the Earthquake in Chile

In some ways, the impact of the recent earthquake in Chile wasn't as immediately obvious as that of the earthquake in Haiti. President Michelle Bachelet initially refused foreign aid, but as the death toll has grown, the need for an international response has become obvious. This week, the American Red Cross increased its original response pledge to a quarter of a million dollars.

read more

February 28 2010

08:24

THE PACIFIC RING OF FIRE

_46474940_ring_of_fire

Like The Economist maps and charts, the BBC uses simple and clear infographics that tell the story.

_47386926_chile_quake_466

February 27 2010

16:41

THE NEW YORK TIMES COVERING THE EARTHQUAKE IN CHILE…FROM RIO DE JANEIRO

2010-02-27_1621

They are called “regional correspondents”.

What a shame!

The NewYork Times covers Chile… from Brazil.

Well the distance between Rio and Santiago is only 1814.31 miles…

It’s like covering Spain from Russia.

That’s the first problem.

The second is the second class content of the dispatch from Rio.

Alexei Barrionuevo reports quoting… Associated Press, Chile’s TVN Cable, the Department of Homeland Security, CNN International, Mrs. Bachelet statements, Facebook and Twitter messages, The White House press secretary, Reuters…

So here we have the most important daily newspaper of the world serving a “mix salad” of second hand news where Eric Lipton contributed reporting from Washington, and Charles Newbery from Buenos Aires, Argentina. Alexei Barrionuevo reported from Rio de Janeiro, and Liz Robbins from New York.

You better follow Twitter real witnesses, than these journalism bureaucrats.

08:56

8.8 EARTHQUAKE IN CHILE: THE FIRST REPORTS

_47384298_chile_santiago_concep3_0210

(BBC map)

The quake struck at 0634 GMT about 91km (56 miles) north-east of the city of Concepcion and 317km south-west of the capital, Santiago.

La Tercera is online.

El Mercurio not at this time.

But Twitter is in full speed.

Here in English (QUAKECHILE)

Here in Spanish (TERREMOTOCHILE).

One of the first pictures from Santiago on Twitpic.

69869388

TV Chile is live here.

69867474

A possible Tsunami travel times map.

Here the last tsunami warning.

The breakingnews on CNN.

Older posts are this way If this message doesn't go away, click anywhere on the page to continue loading posts.
Could not load more posts
Maybe Soup is currently being updated? I'll try again automatically in a few seconds...
Just a second, loading more posts...
You've reached the end.

Don't be the product, buy the product!

Schweinderl