|
Source: ONLamp.com The subprocess module provides a consistent interface to creating and working with additional processes. It offers a higher-level interface than some of the other available modules, and is intended to replace functions such as os.system, os.spawn*, os.popen*, popen2.* and commands.*.
Module: subprocessPurpose: Spawn and communicate with additional processes.Python Version: New in 2.4To make it easier to compare subprocess with those other modules, this week I will re-create earlier examples using the functions being replaced.The subprocess module defines one class, Popen() and a few wrapper functions which use that class. Popen() takes several arguments to make it easier to set up the new process, and then communicate with it via pipes. I will concentrate on example code here; for a complete description of the arguments, refer to section 17.1.1 of the library documentation.A Note About PortabilityThe API is roughly the same, but the underlying implementation is slightly different between Unix and Windows. All of the examples shown here were tested on Mac OS X. Your mileage on a non-Unix OS will vary.Running External CommandTo run an external command without interacting with it, such as one would do with os.system(), Use the call() function.import subprocess# Simple commandsubprocess.call('ls -l', shell=True)$ python replace_os_system.pytotal 16-rw-r--r-- 1 dhellman dhellman 0 Jul 1 11:29 __init__.py-rw-r--r-- 1 dhellman dhellman 1316 Jul 1 11:32 replace_os_system.py-rw-r--r-- 1 dhellman dhellman 1167 Jul 1 11:31 replace_os_system.py~And since we set shell=True, shell variables in the command string are expanded:# Command with shell expansionsubprocess.call('ls -l $HOME', shell=True)total 40drwx------ 10 dhellman dhellman 340 Jun 30 18:45 Desktopdrwxr-xr-x 15 dhellman dhellman 510 Jun 19 07:08 Develdrwx------ 29 dhellman dhellman 986 Jun 29 07:44 Documentsdrwxr-xr-x 44 dhellman dhellman 1496 Jun 29 09:51 DownloadedAppsdrwx------ 55 dhellman dhellman 1870 May 22 14:53 Librarydrwx------ 8 dhellman dhellman 272 Mar 4 2006 Moviesdrwx------ 11 dhellman dhellman 374 Jun 21 07:04 Musicdrwx------ 12 dhellman dhellman 408 Jul 1 01:00 Picturesdrwxr-xr-x 5 dhellman dhellman 170 Oct 1 2006 Publicdrwxr-xr-x 15 dhellman dhellman 510 May 12 15:19 Sitesdrwxr-xr-x 5 dhellman dhellman 170 Oct 5 2005 cfxdrwxr-xr-x 4 dhellman dhellman 136 Jan 23 2006 iPod-rw-r--r-- 1 dhellman dhellman 204 Jun 18 17:07 pgadmin.logdrwxr-xr-x 3 dhellman dhellman 102 Apr 29 16:32 tmpReading Output of Another CommandBy passing different arguments for stdin, stdout, and stderr it is possible to mimic the variations of os.popen().Reading from the output of a pipe:print '\nread:'proc = subprocess.Popen('echo "to stdout"', shell=True, stdout=subprocess.PIPE, )stdout_value = proc.communicate()[0]print '\tstdout:', repr(stdout_value)Writing to the input of a pipe:print '\nwrite:'proc = subprocess.Popen('cat -', shell=True, stdin=subprocess.PIPE, )proc.communicate('\tstdin: to stdin\n')Reading and writing, as with popen2:print '\npopen2:'proc = subprocess.Popen('cat -', shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE, )stdout_value = proc.communicate('through stdin to stdout')[0]print '\tpass through:', repr(stdout_value)Separate streams for stdout and stderr, as with popen3:print '\npopen3:'proc = subprocess.Popen('cat -; echo ";to stderr" 1&2', shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, )stdout_value, stderr_value = proc.communicate('through stdin to stdout')print '\tpass through:', repr(stdout_value)print '\tstderr:', repr(stderr_value)Merged stdout and stderr, as with popen4:print '\npopen4:'proc = subprocess.Popen('cat -; echo ";to stderr" 1&2', shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, )stdout_value, stderr_value = proc.communicate('through stdin to stdout\n')print '\tcombined output:', repr(stdout_value)Sample output:read: stdout: 'to stdout\n'write: stdin: to stdinpopen2: pass through: 'through stdin to stdout'popen3: pass through: 'through stdin to stdout' stderr: ';to stderr\n'popen4: combined output: 'through stdin to stdout\n;to stderr\n'All of the above examples assume a limited amount of interaction. The communicate() method reads all of the output and waits for child process to exit before returning. It is also possible to write to and read from the individual pipe handles used by the Popen instance. To illustrate this, I will use this simple echo program which reads its standard input and writes it back to standard output:import syssys.stderr.write('repeater.py: starting\n')while True: next_line = sys.stdin.readline() if not next_line: break sys.stdout.write(next_line) sys.stdout.flush()sys.stderr.write('repeater.py: exiting\n')Make note of the fact that repeater.py writes to stderr when it starts and stops. We can use that to show the lifetime of the subprocess in the next example. The following interaction example uses the stdin and stdout file handles owned by the Popen instance in different ways. In the first example, a sequence of 10 numbers are written to stdin of the process, and after each write the next line of output is read back. In the second example, the same 10 numbers are written but the output is read all at once using communicate().import subprocessprint 'One line at a time:'proc = subprocess.Popen('repeater.py', shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE, )for i in range(10): proc.stdin.write('%d\n' % i) output = proc.stdout.readline() print output.rstrip()proc.communicate()printprint 'All output at once:'proc = subprocess.Popen('repeater.py', shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE, )for i in range(10): proc.stdin.write('%d\n' % i)output = proc.communicate()[0]print outputNotice where the “repeater.py: exiting” lines fall in the output for each loop:$ python interaction.py One line at a time:repeater.py: starting0123456789repeater.py: exitingAll output at once:repeater.py: startingrepeater.py: exiting0123456789Signaling Between ProcessesIn part 4 of the series on the os module I included an example of signaling between processes using os.fork() and os.kill(). Since each Popen instance provides a pid attribute with the process id of the child process, it is possible to do something similar with subprocess. For this example, I will again set up a separate script for the child process to be executed by the parent process.import osimport signalimport timedef signal_usr1(signum, frame): "Callback invoked when a signal is received" pid = os.getpid() print 'Received USR1 in process %s' % pidprint 'CHILD: Setting up signal handler'signal.signal(signal.SIGUSR1, signal_usr1)print 'CHILD: Pausing to wait for signal'time.sleep(5)And now the parent process:import osimport signalimport subprocessimport timeproc = subprocess.Popen('signal_child.py')print 'PARENT: Pausing before sending signal 'time.sleep(1)print 'PARENT: Signaling %s' % proc.pidos.kill(proc.pid, signal.SIGUSR1)And the output should look something like this:$ python signal_parent.pyCHILD: Setting up signal handlerCHILD: Pausing to wait for signalPARENT: Pausing before sending signal PARENT: Signaling 4124Received USR1 in process 4124ConclusionsAs you can see, subprocess can be much easier to work with than fork, exec, and pipes on their own. It provides all of the functionality of the other modules and functions it replaces, and more. The API is consistent for all uses and many of the extra steps of overhead needed (such as closing extra file descriptors, ensuring the pipes are closed, etc.) are “built in” instead of being handled by your code separately.References:Sample codePyMOTW: os (Part 2)PyMOTW: os (Part 4)Technorati Tags:python, PyMOTW
Source: ONLamp.com I’m pleased to bring the Python Module of the Week series to ONLamp. The goal of the series is to cover one of the Python standard library modules each week with a brief description and examples.
I will post one or two of the popular past episodes here, but for the complete archive check out the main feed. Future posts will be posted to both sites.
Source: ONLamp.com In the second and final part of this series, Andrew Sterling Hanenkamp goes under the hood of his single sign-on tool, CAS+, and shows how it actually works. Along the way, there's lots of useful information about the underlying mechanisms that make all SSO solutions purr.
Source: ONLamp.com Writing documentation is a thankless job when you're getting paid for it, so why in the world would people voluntarily do it for free? Andy Oram wondered just that, so he conducted a survey. Now he's here to present the results and some conclusions he's drawn.
Source: ONLamp.com The often public battle between Microsoft and the Open Source community over document standards has been in the news a lot lately. With states and countries choosing to mandate ODF, Microsoft has been doing its best to get the rival OOXML standard adopted. Sam Hiser, Vice President and Director of Business Affairs at the OpenDocument Foundation, presents his reasons why ODF is the way to go.
Source: ONLamp.com Oracle, DB2, and SQL Server all have analytic functions--ways to preserve the details in a record when you use an aggregate. MySQL lacks them, but you can emulate them if you want. Stephane Faroult continues his look at how to do this in the second part of his series.
Source: ONLamp.com Single Sign-On (SSO) authentication is a necessary component of most enterprise infrastructures. No one wants to supply credentials more than once, and centralizing authentication reduces the number of places password data needs to be duplicated. Andrew Sterling Hanenkamp introduces us to CAS+, a single sign-on solution for Jifty in the first part of a two-part series.
Source: ONLamp.com Ruby on Rails may get all the attention as a quick and easy way to implement CRUD-type screens, but there are similar packages available for other languages. Phil Crow shows how Gantry, a Perl-based CRUD generator, saved Christmas.
Source: ONLamp.com Continuing on from the first part of this series, Howard Feldman dives deeper into all the ways you can morph your web pages with a little JavaScript magic. This month he shows us how to swap photos, do tabbed panes, expand and contract tree lists, and do drop-and-drag item ordering.
Source: ONLamp.com In comments on Sci Foo 2007, Tim O’Reilly wondered if giving Larry Wall a research job may have reduced his immediate practical concerns influencing the development of Perl 6.
I interpret the line of thought as “To what degree does the lack of a single well-defined problem to solve influence the concreteness or the abstractness of the solution and its delivery date?” (I won’t claim that that’s Tim’s line of thought, but it’s the question I’ve pondered this past week.)
Constraints are Good
I’m in favor of constraints.
One of my proudest creations outside of programming is a 300 word horror story. I was proud to write it. I was prouder to edit it, removing 10% of its length (always a good goal in editing), and then to add just enough new material to return to 300 words. It’s stronger for the editing, and even more impressive that I hit 300 words exactly both times. I even spent a nice afternoon writing a microfiction assignment generator which assembles a random group of constraints into a short daily writing assignment.
You can take constraints too far, though; when was the last time you read a good sestina? When was the last time you read any sestina? (How many of you know what that word even means?)
In creative writing, constraints can be liberating. Take today’s microfiction assignment. “Given the character of a talking frying pan…”, you have a character. You have a skeleton on which to hang a story. You know what a frying pan is and what it does. You have hints as to what it considers important, how it spends its day, and how it might talk. Also, “Your word limit is 240 words,” or a couple of paragraphs. That’s a few sentences. If you fail, you’ve spent a few minutes thinking creatively and writing a few sentences. There’s almost no risk.
Constraints also focus the mind. In my horror story, I had to weigh every word carefully. I knew the story I wanted to tell. I had a distinct theme. I wanted subtlety and irony. I had no room to wander, to circumlocute, or to distract. Every word had to be there for a reason. I had to find lightning rather than capturing lightning bugs and calling that good.
Of course, those constraints are all artificial. I admit that. I chose them on purpose for there benefits.
Constraints in Software
Software projects also have constraints. Some are even artificial, but most aren’t. You probably have customer demands to satisfy. You likely have a limited number of available programmers, a finite amount of time, and exhaustable resources. You probably have to work with other people who have different visions of what’s important and what the eventual successful outcomes might be.
Sometimes these focus the mind. Sometimes they don’t. I’m not sure why it can be difficult to communicate the idea that the pure thought-stuff of software, while much more malleable than physical construction materials, for example, still has peculiar gravities and inertias. Perhaps we programmers are better at demonstrating the magic of one- and two-line changes which produce stunningly good effects than we are the weeks and months of experimentation, thought, and redesign which go into making such achievements possible.
Perhaps we’re bad at managing constraints–even so bad as to recognize them far less frequently than we should.
I really like the observation in Extreme Programming that there are four dials you can manipulate in software projects: Time, Scope, Resources, and Quality. (I avoided the possibility of increasing or decreasing quality in Extreme Programming Pocket Guide, but Jim Shore set me straight on that. Sometimes you don’t need a complete mathematical proof of correctness of your entire system. Sometimes a one-shot prototype with an installer that only works on the one machine you’ll ever install it on suffices.) If you’re running short on one element, you have to adjust the other three dials appropriately. If you have an excess of one element, you can reduce the pressure on the other three dials.
It’s not a guarantee of success, nor a mathematically-precise model of reality, but it’s a good reminder that a Work requires a combination of factors. Someone must finish the work with a given amount of resources. This will require a specific period of time. It will have a measurable level of quality.
Constraints in Volunteer Projects
How does this relate to Perl 6?
Perl 5 has been a practical success. Try imagining the ’90s web without it. (Try imagining the 21st century Internet without it. A lot of projects millions of people rely on would disappear. Constraint #1 is that Perl 6 oughtn’t cannibalize on the success of Perl 5 without being substantially better.
Fortunately, the fanatical devotion to backwards compatibility exhibited by Perl 5 makes it a tempting target sometimes.
Perl 5 has lasted for nearly 13 years. That’s a long time. Fortunately, the language design has enabled evolution in a lot of areas (if imperfectly in some). CPAN is the best example. The internals are probably the worst example. Constraint #2 is that Perl 6 can succeed by enabling evolution in future directions. (Larry likes to say that the killer application for Perl 6 is something that no one has invented yet.)
Perl has always been quirky, even among programming languages. Plenty of languages used sigils before (some well and some very badly), but Perl hooked them into the type system. (With that, I’ve confused just about everyone in the world. Think about it.) Even if you throw out the pro-golf, pro-obfuscation coterie (and you really should–just smile and nod as you back away while keeping your hands in plain sight), there’s still a discernable sense of Perlishness in well-written Perl code. Constraint #3 is that Perl 6 should retain that sense of Perlishness, whatever other changes occur.
Some constraints are less philosophical.
By my calculations, Perl 6 has had just under three fully-funded person-years of development. That’s 6000 hours, split between Larry, Damian, Dan, and a handfull of microgrant recipients.
The 2000 developers on Microsoft’s Vista spent that much time reading e-mail in any given week.
Constraint #4 is a staggering reliance on volunteer labor. I have nothing but praise for the volunteers on various Perl 6 projects. I merely mean that practical matters like working at my day job to pay the mortgage, spending time with my family, and sleeping have sometimes taken priority over writing just one more line of code–and I’m not the only developer who’s made that choice occasionally.
It would be nice if there were a pool of Perl-friendly C developers ready to wade into a codebase like Parrot’s. Likewise, Pugs would benefit from the attention of a handful of brilliant Haskell hackers. Yet no matter how easy either project makes it to donate an hour a week, no matter how beautiful the code or clear the requirements, someone still has to sit down to the glow of a text editor and write one line of code at a time.
Plenty of quotes describe research and inspiration as long, dry desert treks punctuated occasionally by brilliant flashes of insight by cool oasis pools. Sometimes you have to slog down several wrong-way dark alleys before you realize that the correct path lies elsewhere. Constraint #5 is that, when inventing the future, sometimes you have to wait for the future to catch up to you. It took the widespread availability of certain features of the Haskell programming language (and some GHC enhancements) to make Pugs possible. The invention of PGE and TGE within Parrot made writing compilers much easier, much as the integration of IMCC made programming Parrot simpler as the development of NQP should make the Perl 6 compiler even that much easier to write.
Sometimes you have to stop to build your tools before you can build your machines.
Embracing Constraints
Kicking against the goads is a good way to end up with a limp. Sometimes the only way to make progress on an impossible task is to realize exactly how impossible it is and to sneak up on it from a different direction.
That’s why there are parallel implementations of Perl 6, for example.
That’s why Parrot generates a lot of C code that we could write (and worse, maintain) by hand.
That’s why the Parrot compiler toolkit is so important. Sure, lex and yacc are good tools. They’ve been good tools for decades, if you want to write another compiler in C.
Sometimes, solving difficult problems requires brilliant new ideas. Sometimes those new ideas are stepwise refinements of older, good ideas. Maybe they’re radical rethinkings in other ways.
Still, the best way I know of to get something done even in the face of constraints–artificial or otherwise–is to accept them as fait accompli, then ask “What if I really didn’t need a thousand words to tell that story? What if a hundred really would suffice?” and then just work toward that goal.
Maybe you’ll have to turn the Scope dial way down, or the Time goal way up. That’s the nice thing about software–and words. It’s almost infinitely malleable.
|
