live output from subprocess command
I'm using a python script as a driver for a hydrodynamics code. When it comes time to run the simulation, I use
subprocess.Popen to run the code, collect the output from stdout and stderr into a
subprocess.PIPE --- then I can print (and save to a log-file) the output information, and check for any errors. The problem is, I have no idea how the code is progressing. If I run it directly from the command line, it gives me output about what iteration its at, what time, what the next time-step is, etc.
Is there a way to both store the output (for logging and error checking), and also produce a live-streaming output?
The relevant section of my code:
ret_val = subprocess.Popen( run_command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True ) output, errors = ret_val.communicate() log_file.write(output) print output if( ret_val.returncode ): print "RUN failed\n\n%s\n\n" % (errors) success = False if( errors ): log_file.write("\n\n%s\n\n" % errors)
Originally I was piping the
tee so that a copy went directly to the log-file, and the stream still output directly to the terminal -- but that way I can't store any errors (to my knowlege).
ret_val = subprocess.Popen( run_command, stdout=log_file, stderr=subprocess.PIPE, shell=True ) while not ret_val.poll(): log_file.flush()
then, in another terminal, run
tail -f log.txt (s.t.
log_file = 'log.txt').
You have two ways of doing this, either by creating an iterator from the
readline functions and do:
import subprocess import sys with open('test.log', 'w') as f: process = subprocess.Popen(your_command, stdout=subprocess.PIPE) for c in iter(lambda: process.stdout.read(1), ''): # replace '' with b'' for Python 3 sys.stdout.write(c) f.write(c)
import subprocess import sys with open('test.log', 'w') as f: process = subprocess.Popen(your_command, stdout=subprocess.PIPE) for line in iter(process.stdout.readline, ''): # replace '' with b'' for Python 3 sys.stdout.write(line) f.write(line)
Or you can create a
reader and a
writer file. Pass the
writer to the
Popen and read from the
import io import time import subprocess import sys filename = 'test.log' with io.open(filename, 'wb') as writer, io.open(filename, 'rb', 1) as reader: process = subprocess.Popen(command, stdout=writer) while process.poll() is None: sys.stdout.write(reader.read()) time.sleep(0.5) # Read the remaining sys.stdout.write(reader.read())
This way you will have the data written in the
test.log as well as on the standard output.
The only advantage of the file approach is that your code doesn't block. So you can do whatever you want in the meantime and read whenever you want from the
reader in a non-blocking way. When you use
readline functions will block until either one character is written to the pipe or a line is written to the pipe respectively.