os.system and subprocess odd behavior

P

py_genetic

Example of the issue for arguments sake:

Platform Ubuntu server 12.04LTS, python 2.7

Say file1.txt has "hello world" in it.

subprocess.Popen("cat < file1 > file2", shell = True)
subprocess.call("cat < file1 > file2", shell = True)
os.system("cat < file1 > file2")


I'm finding that file2 IS created, but with 0bytes in it, this happens when I try any sort of cmd to the system of the nature where I'm putting the output into a file.

I've made sure it isn't a permission issue. The command runs fine from the cmd line and python is being run with super user privileges. Strait from the terminal I get a hello world copy as file2... as expected.

I would like python to simply exec the cmd and move on.... I don't want to read and write the stdout ect into python and write it to a file. Any thoughts as to why this creates file2, but no data appears? Is there a better way to do this?

Thank you!
 
S

Steven D'Aprano

Example of the issue for arguments sake:

Platform Ubuntu server 12.04LTS, python 2.7

Say file1.txt has "hello world" in it.

subprocess.Popen("cat < file1 > file2", shell = True)
subprocess.call("cat < file1 > file2", shell = True)
os.system("cat < file1 > file2")


I'm finding that file2 IS created, but with 0bytes in it, this happens
when I try any sort of cmd to the system of the nature where I'm putting
the output into a file.
Click to expand...

I cannot confirm this behaviour. It works for me. When I try it, all
three code snippets work as expected:

[steve@ando ~]$ rm file2 file3 file4
[steve@ando ~]$ cat file1
hello world
[steve@ando ~]$ python
Python 2.7.2 (default, May 18 2012, 18:25:10)
[GCC 4.1.2 20080704 (Red Hat 4.1.2-52)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
py> import os, subprocess
py> subprocess.Popen("cat < file1 > file2", shell = True)
<subprocess.Popen object at 0xb7e8a62c>
py> subprocess.call("cat < file1 > file3", shell = True)
0
py> os.system("cat < file1 > file4")
0
py> quit()
[steve@ando ~]$ cat file2
hello world
[steve@ando ~]$ cat file3
hello world
[steve@ando ~]$ cat file4
hello world


I have run this multiple times, as an unprivileged user, as the root
user, and as sudo. It works perfectly every time.

Please check your code. Perhaps you have over-simplified the problem.
 
D

Dieter Maurer

py_genetic said:
Example of the issue for arguments sake:

Platform Ubuntu server 12.04LTS, python 2.7

Say file1.txt has "hello world" in it.
Click to expand...
^^^^^^^^^
Here, you speak of "file1.txt" (note the extension ".txt")
subprocess.Popen("cat < file1 > file2", shell = True)
subprocess.call("cat < file1 > file2", shell = True)
os.system("cat < file1 > file2")
Click to expand...

But in your code, you use "file1" (without extension).

If your code really references a non-existing file, you may well
get what you are observing.
 
P

py_genetic

Thanks for verifying this for me Steven. I'm glad you are seeing it work. It's really the strangest thing.

The issue seems to be with the " > outfile.txt" portion of the command.

The actual command is running a query on a verticalDB and dumping the result. The EXACT command run from the command line runs just fine.

Even if I use the simple cat command to and out file as just a simple test case... The file is created with zero bytes (see below)... but its as if python moves on or gets an 0 exit code after the first part of the cmd is executed and no data is written.

-rw-r--r-- 1 root root 0 Dec 14 15:33 QUAD_12142012203251.TXT

Any thoughts as to why on my end this may happen?

Thanks again!
 
O

Oscar Benjamin

Thanks for verifying this for me Steven. I'm glad you are seeing it work.. It's really the strangest thing.

The issue seems to be with the " > outfile.txt" portion of the command.

The actual command is running a query on a verticalDB and dumping the result. The EXACT command run from the command line runs just fine.

Even if I use the simple cat command to and out file as just a simple test case... The file is created with zero bytes (see below)... but its as ifpython moves on or gets an 0 exit code after the first part of the cmd is executed and no data is written.

-rw-r--r-- 1 root root 0 Dec 14 15:33 QUAD_12142012203251.TXT

Any thoughts as to why on my end this may happen?
Click to expand...

Because of the root permissions on the file? What happens if you write
to a file that doesn't need privileged access?

Instead of running the "exact command", run the cat commands you
posted (that Steven has confirmed as working) and run them somewhere
in your user directory without root permissions.

Also you may want to use subprocess.check_call as this raises a Python
error if the command returns an error code.


Oscar
 
P

py_genetic

Oscar, seems you may be correct. I need to run this program as a superuser.. However, after some more tests with simple commands... I seem to be working correctly from any permission level in python.... Except for the output write command from the database to a file. Which runs fine if I paste itinto the cmd line. Also, subprocess.call_check() returns clean. However,nothing is written to the output file when called from python.

so this cmd runs great from the cmd line (sudo or no) however the output file in this case is owned by the sysadmin either way... not root?

/usr/local/Calpont/mysql/bin/mysql --defaults-file=/usr/local/Calpont/mysql/my.cnf -u root myDB < /home/myusr/jobs/APP_JOBS/JOB_XXX.SQL > /home/myusr/jobs/APP_JOBS/JOB_XXX.TXT


When run from sudo python (other files are also created and owned by root correctly) however no output is written from the db command zero byte file only (owned by root)... returns to python with no errors.

I'm sorta at a loss. I'd rather still avoid having python connect to the db directly or reading the data from stdout, is a waste or mem and time for what I need.

Thanks for any more thoughts.
 
P

py_genetic

Oscar, seems you may be correct. I need to run this program as a superuser.. However, after some more tests with simple commands... I seem to be working correctly from any permission level in python.... Except for the output write command from the database to a file. Which runs fine if I paste itinto the cmd line. Also, subprocess.call_check() returns clean. However,nothing is written to the output file when called from python.

so this cmd runs great from the cmd line (sudo or no) however the output file in this case is owned by the sysadmin either way... not root?

/usr/local/Calpont/mysql/bin/mysql --defaults-file=/usr/local/Calpont/mysql/my.cnf -u root myDB < /home/myusr/jobs/APP_JOBS/JOB_XXX.SQL > /home/myusr/jobs/APP_JOBS/JOB_XXX.TXT


When run from sudo python (other files are also created and owned by root correctly) however no output is written from the db command zero byte file only (owned by root)... returns to python with no errors.

I'm sorta at a loss. I'd rather still avoid having python connect to the db directly or reading the data from stdout, is a waste or mem and time for what I need.

Thanks for any more thoughts.
 
O

Oscar Benjamin

Oscar, seems you may be correct. I need to run this program as a superuser. However, after some more tests with simple commands... I seem to be working correctly from any permission level in python.... Except for the output write command from the database to a file. Which runs fine if I paste it into the cmd line. Also, subprocess.call_check() returns clean. However, nothing is written to the output file when called from python.

so this cmd runs great from the cmd line (sudo or no) however the output file in this case is owned by the sysadmin either way... not root?

/usr/local/Calpont/mysql/bin/mysql --defaults-file=/usr/local/Calpont/mysql/my.cnf -u root myDB < /home/myusr/jobs/APP_JOBS/JOB_XXX.SQL > /home/myusr/jobs/APP_JOBS/JOB_XXX.TXT


When run from sudo python (other files are also created and owned by rootcorrectly) however no output is written from the db command zero byte file only (owned by root)... returns to python with no errors.

I'm sorta at a loss. I'd rather still avoid having python connect to thedb directly or reading the data from stdout, is a waste or mem and time for what I need.
Click to expand...

Follow through the bash session below

$ cd /usr
$ ls
bin games include lib local sbin share src
$ touch file
touch: cannot touch `file': Permission denied
$ sudo touch file
[sudo] password for oscar:
$ ls
bin file games include lib local sbin share src
$ cat < file > file2
bash: file2: Permission denied
$ sudo cat < file > file2
bash: file2: Permission denied
$ sudo cat < file > file2
bash: file2: Permission denied
$ sudo cat < file | tee file2
tee: file2: Permission denied
$ sudo cat < file | sudo tee file2
$ ls
bin file file2 games include lib local sbin share src

The problem is that when you do

$ sudo cmd > file2

it is sort of like doing

$ sudo cmd | this_bash_session > file2

so the permissions used to write to file2 are the same as the bash
session rather than the command cmd which has root permissions. By
piping my output into "sudo tee file2" I can get file2 to be written
by a process that has root permissions.

I suspect you have the same problem although it all complicated by the
fact that everything is a subprocess of Python. Is it possibly the
case that the main Python process does not have root permissions but
you are using it to run a command with sudo that then does have root
permissions?

Does piping through something like "sudo tee" help?


Oscar
 
P

photonymous

I hope I understand the question... but shouldn't you wait for the process to complete before exiting?

Something like:

pid = subprocess.Popen(...)
pid.wait()

Otherwise, it'll exit before the background process is done.
 
H

Hans Mulder

I hope I understand the question... but shouldn't you wait for the process to complete before exiting?

Something like:

pid = subprocess.Popen(...)
pid.wait()

Otherwise, it'll exit before the background process is done.
Click to expand...

Why would that be a problem?

-- HansM
 
H

Hans Mulder

Because you don't want to bog the system down with a zombie task.
Click to expand...

I think you're confusing zombies with orphans. A zombie results when
a child exits before the parent and the parent fails to wait for it.
An orphan is what you get if the parent exits before the child does.

On a Un*x system, the 'init' process (the process with ID 1) inherits
orphan processes and collects their exit status when they terminate.
Thus orphans do not become zombies.

I have no idea what would happen under Windows.


Hope this helps,

-- HansM
 
P

py_genetic

Oscar I can confirm this behavior from terminal.

AND this works as well, simulating exactly what I'm doing permissions wise, and calling sudo python test.py below

f1 = open('TESTDIR/file1.txt', 'w')
f1.write('some test here\n')
f1.close()

cmd1 = 'cat < TESTDIR/file1.txt > TESTDIR/file2.txt'
P = Popen(cmd1, shell=True)
P.wait()

cmd2 = 'cat < TESTDIR/file1.txt | sudo tee TESTDIR/file3.txt'
P = Popen(cmd2, shell=True)
P.wait()

-rw-r--r-- 1 root root 15 Dec 18 12:57 file1.txt
-rw-r--r-- 1 root root 15 Dec 18 12:57 file2.txt
-rw-r--r-- 1 root root 15 Dec 18 12:57 file3.txt

HOWEVER...

when using this command from before.... no dice

/usr/local/Calpont/mysql/bin/mysql --defaults-file=/usr/local/Calpont/mysql/my.cnf -u root myDB < /home/myusr/jobs/APP_JOBS/JOB_XXX.SQL > /home/myusr/jobs/APP_JOBS/JOB_XXX.TXT

OR

/usr/local/Calpont/mysql/bin/mysql --defaults-file=/usr/local/Calpont/mysql/my.cnf -u root myDB < /home/myusr/jobs/APP_JOBS/JOB_XXX.SQL | sudo tee /home/myusr/jobs/APP_JOBS/JOB_XXX.TXT

So it's basically as if python gets a response instantly (perhaps from the query) and closes the process, since we've verified its not permissions related.

Perhaps someone can try a mysql cmd line such as above within python? And see if you can verify this behavior. I believe the query returning with no errors is shutting the sub shell/process?

I've tried this with all options p.wait() ect as well as parsing the command and running shell false.

Again the exact command run perfect when pasted and run from the shell. I'll try running it a few other ways with some diff db options.

Follow through the bash session below



$ cd /usr

$ ls

bin games include lib local sbin share src

$ touch file

touch: cannot touch `file': Permission denied

$ sudo touch file

[sudo] password for oscar:

$ ls

bin file games include lib local sbin share src

$ cat < file > file2

bash: file2: Permission denied

$ sudo cat < file > file2

bash: file2: Permission denied

$ sudo cat < file > file2

bash: file2: Permission denied

$ sudo cat < file | tee file2

tee: file2: Permission denied

$ sudo cat < file | sudo tee file2

$ ls

bin file file2 games include lib local sbin share src



The problem is that when you do



$ sudo cmd > file2



it is sort of like doing



$ sudo cmd | this_bash_session > file2



so the permissions used to write to file2 are the same as the bash

session rather than the command cmd which has root permissions. By

piping my output into "sudo tee file2" I can get file2 to be written

by a process that has root permissions.



I suspect you have the same problem although it all complicated by the

fact that everything is a subprocess of Python. Is it possibly the

case that the main Python process does not have root permissions but

you are using it to run a command with sudo that then does have root

permissions?



Does piping through something like "sudo tee" help?





Oscar
Click to expand...
 
P

py_genetic

Oscar I can confirm this behavior from terminal.

AND this works as well, simulating exactly what I'm doing permissions wise, and calling sudo python test.py below

f1 = open('TESTDIR/file1.txt', 'w')
f1.write('some test here\n')
f1.close()

cmd1 = 'cat < TESTDIR/file1.txt > TESTDIR/file2.txt'
P = Popen(cmd1, shell=True)
P.wait()

cmd2 = 'cat < TESTDIR/file1.txt | sudo tee TESTDIR/file3.txt'
P = Popen(cmd2, shell=True)
P.wait()

-rw-r--r-- 1 root root 15 Dec 18 12:57 file1.txt
-rw-r--r-- 1 root root 15 Dec 18 12:57 file2.txt
-rw-r--r-- 1 root root 15 Dec 18 12:57 file3.txt

HOWEVER...

when using this command from before.... no dice

/usr/local/Calpont/mysql/bin/mysql --defaults-file=/usr/local/Calpont/mysql/my.cnf -u root myDB < /home/myusr/jobs/APP_JOBS/JOB_XXX.SQL > /home/myusr/jobs/APP_JOBS/JOB_XXX.TXT

OR

/usr/local/Calpont/mysql/bin/mysql --defaults-file=/usr/local/Calpont/mysql/my.cnf -u root myDB < /home/myusr/jobs/APP_JOBS/JOB_XXX.SQL | sudo tee /home/myusr/jobs/APP_JOBS/JOB_XXX.TXT

So it's basically as if python gets a response instantly (perhaps from the query) and closes the process, since we've verified its not permissions related.

Perhaps someone can try a mysql cmd line such as above within python? And see if you can verify this behavior. I believe the query returning with no errors is shutting the sub shell/process?

I've tried this with all options p.wait() ect as well as parsing the command and running shell false.

Again the exact command run perfect when pasted and run from the shell. I'll try running it a few other ways with some diff db options.

Follow through the bash session below



$ cd /usr

$ ls

bin games include lib local sbin share src

$ touch file

touch: cannot touch `file': Permission denied

$ sudo touch file

[sudo] password for oscar:

$ ls

bin file games include lib local sbin share src

$ cat < file > file2

bash: file2: Permission denied

$ sudo cat < file > file2

bash: file2: Permission denied

$ sudo cat < file > file2

bash: file2: Permission denied

$ sudo cat < file | tee file2

tee: file2: Permission denied

$ sudo cat < file | sudo tee file2

$ ls

bin file file2 games include lib local sbin share src



The problem is that when you do



$ sudo cmd > file2



it is sort of like doing



$ sudo cmd | this_bash_session > file2



so the permissions used to write to file2 are the same as the bash

session rather than the command cmd which has root permissions. By

piping my output into "sudo tee file2" I can get file2 to be written

by a process that has root permissions.



I suspect you have the same problem although it all complicated by the

fact that everything is a subprocess of Python. Is it possibly the

case that the main Python process does not have root permissions but

you are using it to run a command with sudo that then does have root

permissions?



Does piping through something like "sudo tee" help?





Oscar
Click to expand...
 
O

Oscar Benjamin

Can you trim content and interleave your response (instead of
top-posting) please?

HOWEVER...

when using this command from before.... no dice

/usr/local/Calpont/mysql/bin/mysql --defaults-file=/usr/local/Calpont/mysql/my.cnf -u root myDB < /home/myusr/jobs/APP_JOBS/JOB_XXX.SQL > /home/myusr/jobs/APP_JOBS/JOB_XXX.TXT

OR

/usr/local/Calpont/mysql/bin/mysql --defaults-file=/usr/local/Calpont/mysql/my.cnf -u root myDB < /home/myusr/jobs/APP_JOBS/JOB_XXX.SQL | sudo tee /home/myusr/jobs/APP_JOBS/JOB_XXX.TXT

So it's basically as if python gets a response instantly (perhaps from the query) and closes the process, since we've verified its not permissions related.
Click to expand...

I wouldn't say that this is verified.

Have you verified that without the last redirection (to the output
file) you can correctly read the stdout from within Python?

Does the same command work if you put it in a shell script? If so what
about making a script like so:

#!/usr/bin/env bash

jobname="$1"

/usr/local/Calpont/mysql/bin/mysql
--defaults-file=/usr/local/Calpont/mysql/my.cnf -u root myDB <
"/home/myusr/jobs/APP_JOBS/JOB_${jobname}.SQL" | sudo tee
"/home/myusr/jobs/APP_JOBS/JOB_${jobname}.TXT"

If that works when you run it directly, does it work if you do:
subprocess.check_call(['myscript.sh', jobname])


Oscar
 
P

py_genetic

Solved the issue, by injecting the query into the cmd line. Shell script worked fine as if I was cutting and pasting to the prompt. Seems to still be something with the subprocess receiving and exit code before or when the query finishes, just when I ask to to read from the .SQL file.

example called from in python:
mysql .... < file.txt > out.txt <---- doesn't work (query is run 0Byte output)
mysql .... -e "my query" > out.txt <----- does work

However this isn't standard mysql as it's infinidb. Maybe this is an esoteric issue.

Thanks for the help Oscar. Frustrating since it seems illogical.... seems if the cmd runs in the shell it should have the exact same behavior from a subprocess shell=True cmd string call.

If I find anything else I'll update this.
 
P

py_genetic

Solved the issue, by injecting the query into the cmd line. Shell script worked fine as if I was cutting and pasting to the prompt. Seems to still be something with the subprocess receiving and exit code before or when the query finishes, just when I ask to to read from the .SQL file.

example called from in python:
mysql .... < file.txt > out.txt <---- doesn't work (query is run 0Byte output)
mysql .... -e "my query" > out.txt <----- does work

However this isn't standard mysql as it's infinidb. Maybe this is an esoteric issue.

Thanks for the help Oscar. Frustrating since it seems illogical.... seems if the cmd runs in the shell it should have the exact same behavior from a subprocess shell=True cmd string call.

If I find anything else I'll update this.
 
H

Hans Mulder

/usr/local/Calpont/mysql/bin/mysql --defaults-file=/usr/local/Calpont/mysql/my.cnf -u root myDB < /home/myusr/jobs/APP_JOBS/JOB_XXX.SQL > /home/myusr/jobs/APP_JOBS/JOB_XXX.TXT
Click to expand...

If you're trying to interact with a MySQL database, then
you should really use the myslqdb module. Trying to parse
the output of the command-line utility is harder, and is
too fragile for production use.


Hope this helps,

-- HansM
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Similar Threads

strange subprocess behavior when calling ps 15
How to print stdout before writing stdin using subprocess module in Python? 2
how to use subprocess to execute an exe with args and an output 5
help needed with subprocess, pipes and parameters 0
supplying password to subprocess.call('rsync ...'), os.system('rsync ...') 8
subprocess (spawned by os.system) inherits open TCP/UDP/IP port 0
uniicode and executing a process with subprocess.call, or os.system 1
subprocess escaping POpen?! 0

Members online

No members online now.
Total: 33 (members: 1, guests: 32)
Robots: 333

Forum statistics

Threads
473,755
Messages
2,569,536
Members
45,009
Latest member
GidgetGamb

Latest Threads

Top