Scripts occasionally need to execute commands on another machine or copy files to and from remote computer systems. Traditional tools for remote login and file transfer are telnet and ftp, and Python offers the modules telnetlib and ftplib for automating remote login and file transfer via the telnetandftpprotocols. However, many computer systems today deny con- nection through telnet and ftp. These sites must then be accessed by the Secure Shell utilities ssh for remote login and scp or sftp for file transfer.
Inside a script, one can call upsshandscpas system commands or use mod- ules which offer a programming interface9to these tools. We shall stick to the former strategy in the examples here, because my practical experience indi- cates that the stand-alone applicationssshandscpwork more smoothly than their programmable counterparts. Thesshandscp tools will be exemplified in Chapter 8.5.1.
Chapter 8.5.2 presents a script for running a numerical simulation on a remote machine and creating visualizations on the local computer. The tools sshandscptools are used for remote login and file transfer. This is a simple generalization of the simulation and visualization example in Chapter 2.3 to a distributed computing environment. Some comments on “true” distributed computing, through client/server programming, appear in Chapter 8.5.3.
8.5.1 Secure Shell Commands
Remote Host Login. The Secure Shell programssh is used to login to a remote computer over a network. The program prompts you for a password, whereas the login name and the machine name are given as command-line
9 Seedoc.htmllink to the Vaults of Parnassus, then follow link to “Networking”.
Perl has even more utilities for connecting to remote hosts.
8.5. Scripts Involving Local and Remote Hosts 379 arguments. To log in as hpl on ella.simula.no, I can write the operating system commands
ssh -l hpl ella.simula.no or
ssh hpl@ella.simula.no
It can be convenient to define an environment variable (say) rmt as an ab- breviation for the remote host accounthpl@ella.simula.no. Logging on and printing a filerep1.psin the subdirectorydocofhpl’s home directory on the Linux machineella.simula.no can be compactly carried out as follows:
ssh $rmt ’cd doc; lpr rep1.ps’
TheDISPLAY variable is normally transferred byssh, and if not, runssh -X. This mean that X graphics generated on the remote host can be displayed on the local screen, provided you have authorized connection by anxhost $rmt command on the local computer.
Copying Files to a Remote Host. The scpprogram is a Secure Shell coun- terpart tocpfor copying files to and from a remote computer system:
scp bump.ps hpl@$rmt:papers/fluid
This command copies the localbump.ps file to thepapers/fluid directory in hpl’s home directory on$rmt. Here are some other examples involvingscp:
scp ${rmt}:doc/proc/ideas.html . # copy single file scp ${rmt}:doc/proc/ideas\*.html . # copy several files
scp -r doc ${rmt}:doc # recursive copy of directories You can also transfer data usingsftp, which is the Secure Shell version of the widespreadftpprogram. Thesftpprogram allows non-interactive execution by placing the commands in a batch file.
Transfer of a possibly large set of files in directory trees can be done in several ways:
– scp -r copies all files in a directory tree recursively.
– ncftp (a flexible interface to ftp) copies directories recursively by the get -R command.
– tarin combination withfindcan pack selected files from a directory tree in a single file (“tarball”) to be transferred byscpor sftp.
– Python’s tarfile or zipfile modules combined with os.path.walk con- stitute an alternative to Unixtarandfind.
– Thersyncprogram is a useful alternative toscp -r, where only those files that have been changed since the last file transfer are actually copied over the network10.
10 rsyncis particularly well suited for backup or synchronizing directory trees.
Remote Host Connection without Giving a Password. By default, bothssh andscpprompts you for a password. Logging on withsshand copying with scp can also be done in a secure way without providing passwords inter- actively, if you have gone through an authorization procedure between the local and the remote machine. This procedure depends on the version of ssh. Some guidelines on how to set up a password-free connection are listed in doc/ssh-no-password.html (see Chapter 1.2 for how to download the doc directory).
An alternative is to use pexpect to send passwords to the ssh and scp applications automatically, see the end of the next section.
8.5.2 Distributed Simulation and Visualization
Scripts used to automate numerical simulation and visualization, as exempli- fied in Chapter 2.3, often need to perform the simulation and visualization on different computers. We may want to run the heavy numerics on a dedicated, large-scale, parallel machine, and then copy the results to a visualization machine for creating images and movies.
We shall now extend the simviz1.py script from Chapter 2.3 such that it can run the simulations on a remote host. The following modifications of simviz1.py are needed:
– The name of the remote host and the user account we have on this host are introduced as global variables. These variables may be set on the command line by the-hostand -useroptions.
– The commands needed to execute the oscillator program are dumped to a file named run_case.py, where case denotes the case name of the run.
– The run_case.py file together with the input file case.i to oscillator are transferred to the remote host by scp. We store the two files in a subdirectorytmpof the home directory. Thescpcommand can be sketched as
scp run_case.py case.i user@remote_host:tmp
– The simulation is run by executing ansshcommand, typically something like
ssh user@remote_host "cd tmp; python run_case.py"
Therun_case.pyscript makes a new subdirectorycase(and removes the old one, if it exists), moves case.i to the subdirectory, and changes the current working directory to the subdirectorycase. Then theoscillator command is constructed, printed, and executed.
– The result file sim.datis copied from the remote host to the local host.
8.5. Scripts Involving Local and Remote Hosts 381 The command is of the type
scp user@remote_host:tmp/case/sim.dat .
– If everything so far went fine, i.e., the sim.dat file exists in the current working directory on the local host, we proceed with making a Gnuplot script and running Gnuplot, as in the originalsimviz1.py code.
– Finally, we remove the generated filesrun_case.pyandcase.i, as well as sim.dat.
These modifications are quite simple to perform, and the reader can look up all details in the file
src/py/examples/simviz/simviz_ssh.py
Unless you have set up a password free connection between the local and remote host, as mentioned on page 380, all thessh and scp commands will prompt you for a password. If you dislike these prompts you may use the pexpectmodule (see the pexpect link indoc.html) to automatically feed the password. This module can automate dialogs with interactive applications and is very useful. There is an alternative script tosimviz_ssh.pyexemplify- ing the use ofpexpectwhen running the simulation on an account without a password free connection:
src/py/examples/simviz/simviz_ssh_pexpect.py
This code also shows how you can use thegetpass modules to safely ask for passwords in the beginning of a scripts and use it in various ssh and scp commands later.
Exercise 8.24. Using a web site for distributed simulation.
This exercise aims to develop an alternative to thecall_simviz1.pyscript from Chapter 7.2.5. Now we
– fetch user information about parameters on the local host, – generate the case.iinput file for the oscillatorcode, – generate a CGI script to be run on the server,
– upload the input file and CGI script to the server, – run the CGI script on the server,
– retrieve thesim.dat file with result, – generate plots on the local host.
Note that the CGI script can be very simple. The only thing we need to do is to run theoscillator code (all the input from a user is already processed
and available).
8.5.3 Client/Server Programming
The previous section presented the simplest and often also the most stable way of using a remote server for computations, administered by a script on a local client. Nevertheless, using ssh and scp via an operating system call (Chapter 3.1.3) suffers from several shortcomings: (i) a password must be provided for every command, unless the user has an account with a password free connection on the remote host, (ii) communication of data relies on files, (iii) actions on the remote host must be executed as separate scripts, and (iv) the two-way communication must be very limited, otherwise a large number ofsshandscpcommands are needed. Instead, many situations call for a true client–server application, where a client program on the local host can set up a continuous two-way communication with a program on a remote server.
Python has extensive support for client–server programming. I highly recommend the book by Holden, “Python web Programming” [13], for general information about the topic and examples on using relevant Python modules.
In the next paragraphs, the point is just to notify the reader about what type of functionality that Python offers.
The socket module constitutes the basic tool for running client–server configurations. A server script is written to handle connections by client scripts over a network, and the socket module supports functionality for establishing connections and transferring data. See [13, p. 120] for a quick introduction. Development of specialized distributed simulation and visual- ization applications will normally employ the quite low levelsocket module.
If the remote host allows access by telnet or ftp, the Python modules telnetlib and ftplib can be used to connect to a remote host, issue com- mands on that host, and transfer files back and forth.
File transfer is particularly easy and convenient when the files are acces- sible over the Internet, i.e., as URLs. Theurllibmodule (see Chapter 8.3.5) enables copying or reading such files without any need for accounts with passwords or special hacks to get through firewalls. With CGI scripts on the server, called up by a script on the local host as explained in Chapter 7.2.5, you can perform computations on the remote (Internet) server. Small data sets can be sent to the server through the URL, while larger amounts of data are better collected in files and uploaded through an HTML form, see [5, p. 471] for recipes.
8.5.4 Threads
Threads allow multiple tasks to be performed concurrently. For example, a GUI may work with visualization while the main script continues with cal- culations, or two canvas widgets may display graphics concurrently. Threads are often used in scripts dealing with networks and databases, if the network and database communication can run in parallel with other tasks.
8.5. Scripts Involving Local and Remote Hosts 383 The basic recipe for running a function call myfunc(a,b,c) in a separate thread reads
import threading
t = threading.Thread(target=myfunc, args=(a,b,c)) t.start()
<do other tasks>
if not t.isAlive():
# the myfunc(a,b,c) call is finished
By subclassing Thread we may achieve more detailed control. The subclass skeleton looks like
class MyThread(threading.Thread):
def __init__(self, ...):
threading.Thread.__init__(self)
<initializations>
def run(self):
<implement the tasks to be performed in the thread>
t = MyThread(...)
t.start() # calls t.run()
Here is an example on downloading a file in a thread [2]:
class Download(threading.Thread):
def __init__(self, url, filename):
self.url = url; self.filename = filename threading.Thread.__init__(self)
def run(self):
print ’Fetching’, self.url
urllib.urlretrieve(self.url, self.filename) print self.filename, ’is downloaded’
Suppose we have a script that needs to download large data files from a web site, but that other tasks can be done while waiting for the downloads. The next code segment illustrates how to download the files in separate threads:
files = [Download(’http://www.some.where/data/f1.dat’, ’f1.dat’), Download(’http://www.some.where/data/f2.dat’, ’f2.dat’), Download(’http://www.another.place/res.dat’, ’res.dat’)]
for download in files:
download.start()
<do other tasks>
# is f2.dat downloaded?
if not files[1].isAlive():
if os.path.isfile(files[1].filename):
<process file>
An example on using threads for visualization purposes appears in the demo scriptDemo/tkinter/guido/brownian.pyin the standard Python source code distribution.
Thescitools.miscmodule contains a classBackgroundCommand(with short formBG) for running a function call and storing the return value in a separate thread. The class is handy for putting time-consuming calculations in the background in the interactive Python shell:
>>> from scitools.misc import BackgroundCommand as BG
>>> b=BG(’f’, g.gridloop, ’sin(x*y)-exp(-x*y)’)
>>> b.start()
running f=gridloop(’sin(x*y)-exp(-x*y)’,) in a thread
>>> # continue with other interactive tasks
>>> b.finished True
>>> b.f # result of function call in thread
>>> max(b.f) 3.2