Tumblelog by Soup.io
Newer posts are loading.
You are at the newest post.
Click here to check if anything new just came in.

January 20 2012


How to stop missing the good weekends

The BBC's Michael Fish presenting the weather in the 80s, with a ScraperWiki tractor superimposed over LiverpoolFar too often I get so stuck into the work week that I forget to monitor the weather for the weekend when I should be going off to play on my dive kayaks — an activity which is somewhat weather dependent.

Luckily, help is at hand in the form of the ScraperWiki email alert system.

As you may have noticed, when you do any work on ScraperWiki, you start to receive daily emails that go:

Dear Julian_Todd,

Welcome to your personal ScraperWiki email update.

Of the 320 scrapers you own, and 157 scrapers you have edited, we
have the following news since 2011-12-01T14:51:34:

Histparl MP list - https://scraperwiki.com/scrapers/histparl_mp_list :
  * ran 1 times producing 0 records from 2 pages
  * with 1 exceptions, (XLRDError: Unsupported format, or corrupt file: Expected BOF record; found '<!DOCTYP')

...Lots more of the same

This concludes your ScraperWiki email update till next time.

Please follow this link to change how often you get these emails,
or to unsubscribe: https://scraperwiki.com/profiles/edit/#alerts

The idea behind this is to attract your attention to matters you may be interested in — such as fixing those poor dear scrapers you have worked on in the past and are now neglecting.

As with all good features, this was implemented as a quick hack.

I thought: why design a whole email alert system, with special options for daily and weekly emails, when we already have a scraper scheduling system which can do just that?

With the addition of a single flag to designate a scraper as an emailer (plus a further 20 lines of code), a new fully fledged extensible feature was born.

Of course, this is not counting the code that is in the Wiki part of ScraperWiki.

The default code in your emailer looks roughly like so:

import scraperwiki
emaillibrary = scraperwiki.utils.swimport("general-emails-on-scrapers")
subjectline, headerlines, bodylines, footerlines = emaillibrary.EmailMessageParts("onlyexceptions")
if bodylines:
    print "
".join([subjectline] + headerlines + bodylines + footerlines)

As you can see, it imports the 138 lines of Python from general-emails-on-scrapers, which I am not here to talk about right now.

Using ScraperWiki emails to watch the weather

Instead, what I want to explain is how I inserted my Good Weather Weekend Watcher by polling the weather forecast for Holyhead.

My extra code goes like this:

weatherlines = [ ]
if datetime.date.today().weekday() == 2:  # Wednesday
    url = "http://www.metoffice.gov.uk/weather/uk/wl/holyhead_forecast_weather.html"
    html = urllib.urlopen(url).read()
    root = lxml.html.fromstring(html)
    rows = root.cssselect("div.tableWrapper table tr")
    for row in rows:
        #print lxml.html.tostring(row)
        metweatherline = row.text_content().strip()
        if metweatherline[:3] == "Sat":
            subjectline += " With added weather"
            weatherlines.append("*** Weather warning for the weekend:")
            weatherlines.append("   " + metweatherline)

What this does is check if today is Wednesday (day of the week #2 in Python land), then it parses through the Met Office Weather Report table for my chosen location, and pulls out the row for Saturday.

Finally we have to handle producing the combined email message, the one which can contain either a set of broken scraper alerts, or the weather forecast, or both.

if bodylines or weatherlines:
    if not bodylines:
        headerlines, footerlines = [ ], [ ]   # kill off cruft surrounding no message
    print "
".join([subjectline] + weatherlines + headerlines + bodylines + footerlines)

The current state of the result is:

*** Weather warning for the weekend:
  Mon 5Dec

  7 °C
  33 mph
  47 mph
  Very Good

This was a very quick low-level implementation of the idea with no formatting and no filtering yet.

Email alerts can quickly become sophisticated and complex. Maybe I should only send a message out if the wind is below a certain speed. Should I monitor previous days’ weather to predict whether the sea will be calm? Or I could check the wave heights on the off-shore buoys? Perhaps my calendar should be consulted for prior engagements so I don’t get frustrated by being told I am missing out on a good weekend when I had promised to go to a wedding.

The possibilities are endless and so much more interesting than if we’d implemented this email alert feature in the traditional way, rather than taking advantage of the utterly unique platform that we happened to already have in ScraperWiki.

July 17 2011


Phone-hacking and journalism - Andrew Gilligan: you have to sail close to the wind

Telegraph :: In 2008, I won journalist of the year at the British Press Awards for an investigation into a man called Lee Jasper, a senior aide to the then mayor of London, Ken Livingstone. Using a mass of leaked emails, I found that enormous amounts of City Hall money had disappeared into projects run by Jasper’s friends, with little or nothing to show for it. Jasper resigned, and Livingstone’s re-election campaign was damaged.

[Andrew Gilligan:] I did not steal those emails, nor did I ask anyone else to do so. But after the story was over, Livingstone’s biographer reported that they were obtained by someone in City Hall who accessed Jasper’s computer, using a password he had left on a post-it note by his terminal. You could, I suppose, say they were hacked.

Continue to read Andrew Gilligan, www.telegraph.co.uk

December 10 2009


'Climategate' and the Perils of the Media's Short Attention Span

There is a moment with which all stand-up comics are familiar. It comes when they release their big punchline, sometimes known as the "drop." For the drop, timing is everything. A successful drop means a joke takes off. An unsuccessful drop leaves it flat on its face.

The already-infamous release of climate change emails was a fantastically successful drop. Though the emails themselves date from the late 1990s onwards, their release was perfectly timed to capture the media's attention just before the Copenhagen climate talks -- to achieve maximum impact. And it worked.

Why? Not because they undermined the science of global warming. Only a hardened rump of skeptics still believe the world is not warming (as opposed to the larger number who dispute the causes and implications). Nor because they proved there was a global conspiracy of scientists determined to hide the truth from us (rather than a handful of scientists who might well have been manipulating aspects of their data). Nor to help promote a climate change skeptic think tank, the Global Warming Policy Foundation, which coincidentally launched around the same time.

The release of climate change emails dominated the headlines because it was a good news story. It fit with the basic need of news: to reveal something previously hidden, to uncover alleged wrongdoing, and to cast doubt on a widely held consensus.

Media Not Built to Cover Climate Well

In a larger sense, however, climate change doesn't fit with news' needs. The climate doesn't change on an hour-by-hour or daily basis, but over years and decades. It is theoretically urgent but, for most of us, not immediately apparent. Structurally, mainstream news is not built to cover long-term climate change well.

Mainstream news has a short attention span. Alastair Campbell, Tony Blair's former director of communications, remarked that if a politician could weather a media storm for 10 days then he would survive. The media would move on.

Newspapers -- understandably -- don't want to fill their front pages day in and day out with gradual news. Gradual news doesn't sell. "Antarctic ice cap retreated one foot" is unlikely to get people reaching into their pockets for loose change.

As a result, we tend to get treated either to occasional apocalyptic headlines of "the end is nigh" variety, or to news that bucks the scientific consensus.

So, when something like these emails comes along, the story is irresistible to most news outlets. Not just irresistible to cover, but irresistible to elevate the emails from an indication of shabby scientific behavior by a small number of scientists into evidence of a massive global warming conspiracy.

Contrarians, But About the Wrong Things

The problem is that a lot of those within mainstream media are a little bored of climate change. Journalists don't like consensus, especially not when it is foisted on them by ivory-towered experts on the basis of "trust us, we know more about this than you do." A lot of journalists are contrarians, and, for the most part, this is a very good thing.

But when it comes to climate change, many seem to be misdirecting their contrariness. Rather than being contrary about the science, about which the vast majority of journalists know very little, shouldn't journalists be contrarian about the difficult political implications? Isn't that the territory most of us are going to have to live on for the next 50 odd years? And the territory that most journalists would feel more comfortable inhabiting?

Older posts are this way If this message doesn't go away, click anywhere on the page to continue loading posts.
Could not load more posts
Maybe Soup is currently being updated? I'll try again automatically in a few seconds...
Just a second, loading more posts...
You've reached the end.
Get rid of the ads (sfw)

Don't be the product, buy the product!