7.2 Adding Web Interfaces to Scripts
7.2.5 Using Web Applications from Scripts
The use of forms in web pages is primarily a tool for creating interactive applications. However, such interactive applications do not need a human in the other end. We can in fact let a script automate the communication with the interactive web pages. The main tool for this purpose is theurllib module, which is briefly demonstrated in Chapter 8.3.5. Here we just apply some other functionality from that module.
Automating the Interaction with a Scientific Hello World CGI Script. A simple example may illustrate how we can use a script to call up a web application with some form parameters, retrieve the resulting HTML text, and process the text. Our aim is to write a Scientific Hello World script like src/py/intro/hw.py, but the sine computation is to be carried out by a CGI
script on a web server. For the latter purpose we can use the hw2.py.cgi script from Chapter 7.1.
Thehw2.py.cgiscript processes a form with one field, namedr. The value of this field can be specified as a part of the URL by adding a special encoding of the field name and value:
http://www.some.where/cgi/hw2.py.cgi?r=0.1
In this example we specify therfield to be 0.1. Loading this augmented URL is equivalent to loading
http://www.some.where/cgi/hw2.py.cgi
into a browser, filling the entry field with the number 0.1, and pressing the submit (here named “equals”) button.
The script to be written must open a URL augmented with the form parameter, extract the HTML text, and find the value of the sine computation in the HTML text. The first step concerns encoding the form field values as a part of the URL. To this end, we should use the urllib.urlencoding function. This function takes a dictionary with the form field names as keys and the form field contents as values, and returns an encoded string. Here is an example involving three form fields (p1, p2, andq1) containing strings, numbers, and special characters:
>>> import urllib
>>> p = {’p1’:’5 > 2 is true’,’p2’: 1.0/3, ’q1’: ’B & W’}
>>> params = urllib.urlencode(p)
>>> params
’p2=0.333333333333&q1=B+%26+W&p1=5+%3E+2+is+true’
The URL is now to be augmented by a question mark and theparamsstring:
URL = ’http://www.some.where/cgi/somescript.cgi’
f = urllib.urlopen(URL + ’?’ + params)
This augmented URL corresponds to data transfer by the GET method. The POST method is implied by the call
f = urllib.urlopen(URL, params)
We can now make a script that employs the hw2.py.cgi web application to calculate the sine of a number:
#!/usr/bin/env python
"""Front-end script to hw2.py.cgi."""
import urllib, sys, re r = float(sys.argv[1])
params = urllib.urlencode({’r’: r})
URLroot = ’http://www.ifi.uio.no/~inf3330/scripting/src/py/cgi/’
f = urllib.urlopen(URLroot + ’hw2.py.cgi?’ + params)
# grab s (=sin(r)) from the output HTML text:
for line in f.readlines():
7.2. Adding Web Interfaces to Scripts 315 m = re.search(r’"equalsbutton">(.*)$’, line)
if m:
s = float(m.group(1)); break print ’Hello, World! sin(%g)=%g’ % (r,s)
This complete script is found insrc/py/cgi/call_hw2.py. First, we feed the web page with a numberrread from the command line. Even in this simple example we useurllib.urlencode to encode the form parameter. The result- ing web page, containing the sine of therparameter, is read using the file-like object created byurllib.urlopen. Knowing the structure of the HTML text, we can create a regular expression (see Chapter 8.2) to extract the sine value and store it ins. At the end, we write out a message involvingrands. The script behaves as the basicsrc/py/intro/hw.py script, the only difference is that the sine computation is carried out on a web server.
Distributed Simulation and Visualization. Having seen how to call up the hw2.py.cgi application, we have the tools at hand to construct a more use- ful example. Suppose we have a web application that runs a simulator and creates some graphics. Thesimviz1.py.cgiscript from Chapter 7.2 is a sim- ple application of this kind. Our aim now is to create a front-end script to simviz1.py.cgi. The front-end takes the same command-line arguments as simviz1.pyfrom Chapter 2.3, performs the simulation on a web server, trans- fers the plots back to the local host, and displays the graphics. In other words, the front-end works likesimviz1.py, but the computations are performed on a remote web server.
The first task is to store the relevant command-line information, i.e., the command-line arguments corresponding to field names in the form in simviz1.py.cgi, in a dictionary. The dictionary must then be translated to a proper URL encoding. The next step is to augment the URL with the encoded parameters. The output of the web application is not of primary interest, what we want is the resulting plots. However, the PNG plot file has a filename with a random number, which is unknown to us, so we need to examine the HTML output (typically with the aid of regular expressions) to see what the name of the PNG file is.
The complete script is quite compact and listed below. The script is named call_simviz1.py and found insrc/py/cgi.
#!/usr/bin/env python
"""Front-end script to simviz1.py.cgi."""
import math, urllib, sys, os, re
# load command-line arguments into dictionary of legal form params:
p = {’case’: ’tmp1’, ’m’: 1, ’b’: 0.7, ’c’: 5, ’func’: ’y’,
’A’: 5, ’w’: 2*math.pi, ’y0’: 0.2, ’tstop’: 30, ’dt’: 0.05}
for i in range(len(sys.argv[1:])):
if sys.argv[i] in p:
p[sys.argv[i]] = sys.argv[i+1]
params = urllib.urlencode(p)
URLroot = ’http://www.ifi.uio.no/~inf3330/scripting/src/py/cgi/’
f = urllib.urlopen(URLroot + ’simviz1.py.cgi?’ + params) file = p[’case’] + ’.ps’
urllib.urlretrieve(’%s%s/%s’ % (URLroot,p[’case’],file), file)
# the PNG file has a random number; get the filename from
# the output HTML file of the simviz1.py.cgi script:
for line in f.readlines():
m = re.search(r’IMG SRC="(.*?)"’, line) if m:
file = m.group(1).strip(); break
urllib.urlretrieve(’%s%s/%s’ % (URLroot,p[’case’],file), file) from subprocess import call
call(’display ’ + file, shell=True) # show plot on the screen From these examples you should be able to interact with web applications in scripts.
Remark. After having supplied data to a web form, we retrieve an HTML file.
In our two previous simple examples we could extract relevant information by the HTML code by regular expressions. With more complicated HTML files it is beneficial to interpret the contents with an HTML parser. Python comes with a module htmllib defining a class HTMLParser for this purpose.
Examples on usingHTMLParsercan be found in [3, Ch. 20] and [13, Ch. 5].