How to start new process or run shell commands within python?

Starting new process or running shell commands is a common task in python and there are many ways to excute this process. After google the problem in the web, I found the summary of the “start new process or run shell commands within python” is messy. Following is my summary of this problem in python, which include os.system(), os.popen(), commands module and subprocess module. It references some materials from the web and I will keep it here as a note.

1. os.system(command)

Execute the command (a string) in a subshell. This is implemented by calling the Standard C function system(), and has the same limitations. Changes to sys.stdin, etc. are not reflected in the environment of the executed command.

On Unix, the return value is the exit status of the process encoded in the format specified for wait(). Note that POSIX does not specify the meaning of the return value of the C system() function, so the return value of the Python function is system-dependent.

On Windows, the return value is that returned by the system shell after running command, given by the Windows environment variable COMSPEC: on command.com systems (Windows 95, 98 and ME) this is always 0; on cmd.exe systems (Windows NT, 2000 and XP) this is the exit status of the command run; on systems using a non-native shell, consult your shell documentation.

The subprocess module provides more powerful facilities for spawning new processes and retrieving their results; using that module is preferable to using this function. See the Replacing Older Functions with the subprocess Module section in the subprocess documentation for some helpful recipes.

os.system(command) is the first way to run shell commands within python I found and it is simple:
…
>>> import os
>>> os.system(“md5sum –c file.md5″)
file: OK
0
…
Here the “0” is the return value of os.system(“md5sum –c file”), which is the exit status of the process encoded in the format specified for wait(). Note that POSIX does not specify the meaning of the return value of the C system() function, so the return value of the Python function is system-dependent. But I need the “OK” to notice the python program to execute the next step, so os.system() is not my first choice.

2. os.popen(command[, mode[, bufsize]])

Open a pipe to or from command. The return value is an open file object connected to the pipe, which can be read or written depending on whether mode is ‘r’ (default) or ‘w’. The bufsize argument has the same meaning as the corresponding argument to the built-in open() function. The exit status of the command (encoded in the format specified for wait()) is available as the return value of the close() method of the file object, except that when the exit status is zero (termination without errors), None is returned.

…
>>> ret = os.popen(‘md5sum -c file.md5′)
>>> ret
<open file ‘md5sum -c file.md5′, mode ‘r’ at 0x2b5a79b5cd50>
>>> dir(ret)
['__class__', '__delattr__', '__doc__', '__getattribute__', '__hash__', '__init__', '__iter__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__str__', 'close', 'closed', 'encoding', 'fileno', 'flush', 'isatty', 'mode', 'name', 'newlines', 'next', 'read', 'readinto', 'readline', 'readlines', 'seek', 'softspace', 'tell', 'truncate', 'write', 'writelines', 'xreadlines']
>>> print ret.readline()
file.data: OK
…
It seems that os.popen() can be used by my python program, cause the return value include the “OK” status. But can we have a better choice?

3. commands module — Utilities for running commands

Platforms: Unix

The commands module contains wrapper functions for os.popen() which take a system command as a string and return any output generated by the command and, optionally, the exit status.

The subprocess module provides more powerful facilities for spawning new processes and retrieving their results. Using the subprocess module is preferable to using the commands module.

Note

In Python 3.x, getstatus() and two undocumented functions (mk2arg() and mkarg()) have been removed. Also, getstatusoutput() and getoutput() have been moved to the subprocess module.

The commands module defines the following functions:

commands.getstatusoutput(cmd)
Execute the string cmd in a shell with os.popen() and return a 2-tuple (status, output). cmd is actually run as { cmd ; } 2>&1, so that the returned output will contain output or error messages. A trailing newline is stripped from the output. The exit status for the command can be interpreted according to the rules for the C function wait().

commands.getoutput(cmd)
Like getstatusoutput(), except the exit status is ignored and the return value is a string containing the command’s output.

commands.getstatus(file)

Return the output of ls -ld file as a string. This function uses the getoutput() function, and properly escapes backslashes and dollar signs in the argument.

Deprecated since version 2.6: This function is nonobvious and useless. The name is also misleading in the presence of getstatusoutput().

Example:

>>> import commands
>>> commands.getstatusoutput(‘ls /bin/ls’)
(0, ‘/bin/ls’)
>>> commands.getstatusoutput(‘cat /bin/junk’)
(256, ‘cat: /bin/junk: No such file or directory’)
>>> commands.getstatusoutput(‘/bin/junk’)
(256, ‘sh: /bin/junk: not found’)
>>> commands.getoutput(‘ls /bin/ls’)
‘/bin/ls’
>>> commands.getstatus(‘/bin/ls’)
‘-rwxr-xr-x 1 root 13352 Oct 14 1994 /bin/ls’

For my problem, the commands.getstatusoutput() can meet my needs:
…
>>> import commands
>>> ret1, ret2 = commands.getstatusoutput(‘md5sum -c file.md5′)
>>> print ret1, ret2
0 file: OK
…
But “Deprecated since version 2.6: The commands module has been removed in Python 3.0. Use the subprocess module instead”, so it is time to see the subprocess module.

4. subprocess module— Subprocess management

The subprocess module allows you to spawn new processes, connect to their input/output/error pipes, and obtain their return codes. This module intends to replace several other, older modules and functions, such as:

os.system
os.spawn*
os.popen*
popen2.*
commands.*

This module defines one class called Popen:

class Popen(args, bufsize=0, executable=None,
stdin=None, stdout=None, stderr=None,
preexec_fn=None, close_fds=False, shell=False,
cwd=None, env=None, universal_newlines=False,
startupinfo=None, creationflags=0):

Arguments are:

– args should be a string, or a sequence of program arguments.
The program to execute is normally the first item in the args
sequence or string, but can be explicitly set by using the
executable argument.

On UNIX, with shell=False (default): In this case, the Popen
class uses os.execvp() to execute the child program. args
should normally be a sequence. A string will be treated as a
sequence with the string as the only item (the program to
execute).

On UNIX, with shell=True: If args is a string, it specifies the
command string to execute through the shell. If args is a
sequence, the first item specifies the command string, and any
additional items will be treated as additional shell arguments.

On Windows: the Popen class uses CreateProcess() to execute the
child program, which operates on strings. If args is a
sequence, it will be converted to a string using the
list2cmdline method. Please note that not all MS Windows
applications interpret the command line the same way: The
list2cmdline is designed for applications using the same rules
as the MS C runtime.

– bufsize, if given, has the same meaning as the corresponding
argument to the built-in open() function: 0 means unbuffered, 1
means line buffered, any other positive value means use a buffer
of (approximately) that size. A negative bufsize means to use
the system default, which usually means fully buffered. The
default value for bufsize is 0 (unbuffered).

– stdin, stdout and stderr specify the executed programs’ standard
input, standard output and standard error file handles,
respectively. Valid values are PIPE, an existing file
descriptor (a positive integer), an existing file object, and
None. PIPE indicates that a new pipe to the child should be
created. With None, no redirection will occur; the child’s file
handles will be inherited from the parent. Additionally, stderr
can be STDOUT, which indicates that the stderr data from the
applications should be captured into the same file handle as for
stdout.

– If preexec_fn is set to a callable object, this object will be
called in the child process just before the child is executed.

– If close_fds is true, all file descriptors except 0, 1 and 2
will be closed before the child process is executed.

– If shell is true, the specified command will be executed through
the shell.

– If cwd is not None, the current directory will be changed to cwd
before the child is executed.

– If env is not None, it defines the environment variables for the
new process.

– If universal_newlines is true, the file objects stdout and
stderr are opened as a text file, but lines may be terminated
by any of ‘n’, the Unix end-of-line convention, ‘r’, the
Macintosh convention or ‘rn’, the Windows convention. All of
these external representations are seen as ‘n’ by the Python
program. Note: This feature is only available if Python is
built with universal newline support (the default). Also, the
newlines attribute of the file objects stdout, stdin and stderr
are not updated by the communicate() method.

– The startupinfo and creationflags, if given, will be passed to
the underlying CreateProcess() function. They can specify
things such as appearance of the main window and priority for
the new process. (Windows only)

This module also defines two shortcut functions:

– call(*args, **kwargs):
Run command with arguments. Wait for command to complete,
then return the returncode attribute.

The arguments are the same as for the Popen constructor.
Example:

retcode = call(["ls", "-l"])

For my problem, I use the following code to replace the os.popen():

…
>>> from subprocess import Popen, PIPE
>>> ret = Popen(“md5sum -c file.md5″, shell=True, stdout=PIPE).stdout
>>> dir(ret)
['__class__', '__delattr__', '__doc__', '__getattribute__', '__hash__', '__init__', '__iter__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__str__', 'close', 'closed', 'encoding', 'fileno', 'flush', 'isatty', 'mode', 'name', 'newlines', 'next', 'read', 'readinto', 'readline', 'readlines', 'seek', 'softspace', 'tell', 'truncate', 'write', 'writelines', 'xreadlines']
>>> print ret.read()
file: OK
…

References:
http://docs.python.org/library/os.html
http://docs.python.org/library/commands.html
http://docs.python.org/library/subprocess.html
http://www.python.org/dev/peps/pep-0324/

Post by 52nlp

Related posts:

1. “set -x”: Prints executed commands and their arguments
2. Moses Support Digest: Moses decoder on windows freezes after 5 sentences
3. Moses Support Digest:The results of your email commands
4. “Nohup” and “Screen”
5. Error “join: multi-character tab `\\t’” for using join tab
6. Two methods to redirect output for “set -x”
7. Moses Support Digest:Error compiling on Linux