CHAPTER 6 ■ DATA SOURCES 193 ■ Tip Output each piece of data on a separate line, making it easier for other tools to extract the information. You now have a way of knowing which are the next trains to leave. This could be incorporated into a daily news feed, recited by a speech synthesizer while making breakfast, added to a personal aggregator page, or used to control the alarm clock. (The method for this will be discussed later.) Road Traffic With the whole world and his dog being in love with satellite navigation systems, the role of web-based traffic reports has become less useful in recent years. And with the cost coming down every year, it’s unlikely to gain a resurgence any time soon. However, if you have a choice of just one gadget—a SatNav or a web-capable handheld PC—the latter can still win out with one of the live traffic web sites. The United Kingdom has sites like Frixo (www.frixo.com) that report traffic speed on all major roads and integrate Google Maps so you can see the various hotspots. It also seems like they have thought of the HA market, since much of the data is easily accessible, with clear labels for the road speeds between each motorway junction, with the roadwork locations, and with travel news. Weather Sourcing weather data can occur from three sources: an online provider, a personal weather station, or by looking out of the window! I will consider only the first two in the following sections. Forecasts Although there appear to be many online weather forecasts available on the Web, most stem from the Weather Channel’s own Weather.com. This site provides a web plug-in (www.weather.com/services/downloads) and desktop app (Windows-only, alas) to access its data, but currently there’s nothing more open than that in the way of an API. Fortunately, many of the companies that have bought licenses to this data provide access to it for the visitors to their web site and with fewer restrictions. Yahoo! Weather, for example, has data in an XML format that works well but requires a style sheet to convert it into anything usable. Like the train times you’ve just seen, each site presents what it feels is the best trade-off between information and clarity. Consequently, some weather reports comprise only one-line daily commentaries, while others have an hourly breakdown, with temperatures, wind speed, and windchill factors. Pick one with the detail you appreciate and, as mentioned previously, is available with an API or can easily be scraped. In this example, I’ll use the Yahoo! reports. This is an XML file that changes as often as the weather (literally!) and can be downloaded according to your region. This can be determined by going through the Yahoo! weather site as a human and noting the arguments in the URL. For London, this is UKXX0085, which enables the forecast feed to be downloaded with this: #!/bin/bash LOGFILE=/var/log/minerva/cache/weather.xml wget -q http://weather.yahooapis.com/forecastrss?p=UKXX0085 -O $LOGFILE CHAPTER 6 ■ DATA SOURCES 194 You can then process this with XML using a style sheet and xsltproc: RESULT_INFO=/var/log/minerva/cache/weather_info.txt rm $RESULT_INFO xsltproc /usr/local/minerva/bin/weather/makedata.xsl $LOGFILE > $RESULT_INFO This converts a typical XML like this: <?xml version="1.0" encoding="UTF-8" standalone="yes" ?> <rss version="2.0" xmlns:yweather="http://some_weather_site.com/ns/rss/1.0> <channel> <title>Weather - London, UK</title> <language>en-us</language> <yweather:location city="Luton" region="" country="UK"/> <yweather:units temperature="F" distance="mi" pressure="in" speed="mph"/> <yweather:wind chill="26" direction="50" speed="10" /> <yweather:atmosphere humidity="93" visibility="3.73" pressure="30.65" rising="1"/> <yweather:astronomy sunrise="7:50 am" sunset="4:38 pm"/> <image> <title>Weather</title> <width>142</width> <height>18</height> <url>http://todays_weather_chart.gif</url> </image> <yweather:forecast day="Tue" date="26 Jan 2010" low="30" high="36" text="Mostly Cloudy" code="27" /> <yweather:forecast day="Wed" date="27 Jan 2010" low="26" high="35" text="Partly Cloudy" code="30" /> <guid isPermaLink="false">UKXX0085_2010_01_26_4_20_GMT</guid> </item> </channel> </rss> into text like this: day:Tuesday description:Mostly Cloudy low:30 high:36 end: day:Wednesday description:Partly Cloudy low:26 high:35 end: CHAPTER 6 ■ DATA SOURCES 195 That is perfect for speech output, status reports, or e-mail. The makedata.xsl file, however, is a little more fulsome: <?xml version="1.0" encoding="utf-8"?> <xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:scripts="http://www.bluedust.com/sayweather" xmlns:msxsl="urn:schemas-microsoft-com:xslt" xmlns:yweather="http://xml.weather.yahoo.com/ns/rss/1.0" > <xsl:output method="text" encoding="utf-8" media-type="text/plain"/> <xsl:template match="/"> <xsl:apply-templates select="rss"/> <xsl:apply-templates select="channel"/> </xsl:template> <xsl:template match="channel"> <xsl:apply-templates select="item"/> </xsl:template> <xsl:template match="item"> <xsl:apply-templates select="yweather:forecast"/> </xsl:template> <xsl:template match="yweather:forecast"> <xsl:text>day:</xsl:text> <xsl:if test="@day = 'Mon'"> <xsl:text>Monday</xsl:text> </xsl:if> <xsl:if test="@day = 'Tue'"> <xsl:text>Tuesday</xsl:text> </xsl:if> <xsl:if test="@day = 'Wed'"> <xsl:text>Wednesday</xsl:text> </xsl:if> <xsl:if test="@day = 'Thu'"> <xsl:text>Thursday</xsl:text> </xsl:if> <xsl:if test="@day = 'Fri'"> <xsl:text>Friday</xsl:text> </xsl:if> <xsl:if test="@day = 'Sat'"> <xsl:text>Saturday</xsl:text> </xsl:if> CHAPTER 6 ■ DATA SOURCES 196 <xsl:if test="@day = 'Sun'"> <xsl:text>Sunday</xsl:text> </xsl:if> <xsl:text> description:</xsl:text> <xsl:value-of select="@text"/> <xsl:text> low:</xsl:text> <xsl:value-of select="@low"/> <xsl:text> high:</xsl:text> <xsl:value-of select="@high"/> <xsl:text> end: </xsl:text> </xsl:template> </xsl:stylesheet> In several places, you will note the strange carriage returns included to produce a friendlier output file. Because of the CPU time involved in querying these APIs, you download and process them with a script (like the one shown previously) and store its output in a separate file. In this way, you can schedule the weather update script once at 4 a.m. and be happy that the data will be immediately available if/when you query it. The weatherstatus script then becomes as follows: #!/bin/bash RESULT_INFO=/var/log/minerva/cache/weather_info.txt if [ -f $RESULT_INFO]; then cat $RESULT_INFO exit 0; else echo "No weather data is currently available" exit 1; fi This allows you to pipe the text into speech-synthesized alarm calls, web reports, SMS messages, and so on. There are a couple of common rules here, which should be adopted wherever possible in this and other types of data feed: • Use one line for each piece of data to ease subsequent processing. • Remove the old status file first, because erroneous out-of-date information is worse than none at all. • Don’t store time stamps; the file has those already. • Don’t include graphic links, not all mediums support them. CHAPTER 6 ■ DATA SOURCES 197 In the case of weather reports, you might take exception to the last rule, because it’s nice to have visual images for each of the weather states. In this case, it is easier to adopt two different XML files, targeting the appropriate medium. Minerva does this by having a makedata.xsl for the full report and a simpler sayit.xsl that generates sparse text for voice and SMS. Local Reporting Most gadget and electronic shops sell simple weather stations for home use. These show the temperature, humidity, and atmospheric pressure. All of these, with some practice, can predict the next day’s weather for your specific locale and provide the most accurate forecast possible, unless you live next door to the national weather center! Unfortunately, most of these devices provide no way for it to interface with a computer and therefore with the rest of the world. There are some devices, however, and some free software called wview (www.wviewweather.com) to connect with it. This software is a collection of daemons and tools to read the archive data from a compatible weather station. If the station reports real-time information only, then the software will use an SQL database to create the archive. You can then query this as shown previously to generate your personal weather reports. ■ Note If temperature is your only concern, there are several computer-based temperature data loggers on the market that let you monitor the inside and/or outside temperature of your home. Many of these can communicate with a PC through the standard serial port. Radio Radio has been the poor cousin of TV for so long that many people forget it was once our most important medium, vital to the war effort in many countries. And it’s not yet dead! 5 Nowhere else can you get legally free music, band interviews, news, and dramas all streamed (often without ads) directly to your ears. Furthermore, this content is professionally edited and chosen so that it matches the time of day (or night) at which it’s broadcast. Writing a piece of intelligent software to automatically pick some night-time music is unlikely to choose as well as your local radio DJ. From a technological standpoint, radio is available for free with many TV cards and has simple software to scan for stations with fmscan and tune them in using fm. They usually have to be installed separately from the TV tuning software, however: apt-get install fmtools Knowing the frequencies of the various stations can be achieved by researching your local radio listing magazines (often bundled with the TV guide) or checking the web site for the radio regulatory body in your country, such as the Federal Communications Commission (FCC) in the United States 5 However, amusingly, the web site for my local BBC radio station omits its transmission frequency. . Road Traffic With the whole world and his dog being in love with satellite navigation systems, the role of web-based traffic reports has become less useful in recent years. And with the cost. xmlns:msxsl="urn:schemas-microsoft-com:xslt" xmlns:yweather="http://xml.weather.yahoo.com/ns/rss/1.0" > <xsl:output method="text" encoding="utf-8" media-type="text/plain"/>. is the best trade-off between information and clarity. Consequently, some weather reports comprise only one-line daily commentaries, while others have an hourly breakdown, with temperatures,