Multithreaded python script calls the COMMAND LINE

J

johnny

I have python script does ftp download in a multi threaded way. Each
thread downloads a file, close the file, calls the comman line to
convert the .doc to pdf. Command line should go ahead and convert the
file. My question is, when each thread calls the command line, does one
command line process all the request, or each thread creates a one
command line process for themselves and executes the command? For some
reason I am getting "File '1' is alread exists, Do you want to
overwrite [y/n]?" I have to manually enter 'y' or 'n' for the python
script to complete.
 
G

gagsl-py

I have python script does ftp download in a multi threaded way. Each
thread downloads a file, close the file, calls the comman line to
convert the .doc to pdf. Command line should go ahead and convert the
file. My question is, when each thread calls the command line, does one
command line process all the request, or each thread creates a one
command line process for themselves and executes the command? For some
reason I am getting "File '1' is alread exists, Do you want to
overwrite [y/n]?" I have to manually enter 'y' or 'n' for the python
script to complete.

That depends on how you invoke it: os.system creates a new shell which
in turn creates a new process; the spawn* functions do that directly.
Anyway, if you have many conversion processes running, you should pass
them unique file names to avoid conflicts.
Where does the '1' name come from? If it's you, don't use a fixed name
- the tempfile module may be useful.
 
J

johnny

That depends on how you invoke it: os.system creates a new shell which
in turn creates a new process; the spawn* functions do that directly.

I am using os.system. Here is my code

import ftplib, posixpath, threading
from TaskQueue import TaskQueue

def worker(tq):
while True:
host, e = tq.get()

c = ftplib.FTP(host)
c.connect()
try:
c.login()
p = posixpath.basename(e)
ps_dir = r'H:/ftp_download/'
filename = download_dir+p
fp = open(filename, 'wb')
try: c.retrbinary('RETR %s' % e, fp.write)
finally: fp.close()
finally: c.close()
if (p.lower().endswith('.ps') ):
partFileName = p.split('.', 1)
movedFile = download_dir + p
#movedFile = p
finalFile = ps_dir + partFileName[0]+'.pdf'
encode_cmd = r'ps2pdf '+ movedFile + ' '+ finalFile
os.system(encode_cmd)

tq.task_done()

if __name__ == '__main__':
main()

def main():
q = TaskQueue()
#host = 'ftp.microsoft.com'
host = 'mysite.com'
c = ftplib.FTP(host)
c.connect()
try:
#c.login()
c.login()

#folder = '/deskapps/kids/'
folder = ''
for n in c.nlst(folder):
if n.lower().endswith('.ps'):
q.put((host, n))


finally: c.close()

numworkers = 4
for i in range(numworkers):
t = threading.Thread(target=worker, args=(q,))
t.setDaemon(True)
t.start()

q.join()
print 'Done.'
Anyway, if you have many conversion processes running, you should pass
them unique file names to avoid conflicts.
Where does the '1' name come from? If it's you, don't use a fixed name
- the tempfile module may be useful.

I am giving unique name to the converted file with .pdf. I think
something up with the thread. I am new to python, I am not sure what's
wrong.
 
G

Gabriel Genellina

I am giving unique name to the converted file with .pdf. I think
something up with the thread. I am new to python, I am not sure what's
wrong.

Unless the ps2pdf program uses explicitely '1' as a filename, I don't
see where it may come from. Your python code doesn't generate it -
unless you have a 1.ps file.

[Quoting from a previous message]
For some
reason I am getting "File '1' is alread exists, Do you want to
overwrite [y/n]?" I have to manually enter 'y' or 'n' for the python
script to complete.

Is that '1' just an example, or you always get that?


--
Gabriel Genellina
Softlab SRL

__________________________________________________
Correo Yahoo!
Espacio para todos tus mensajes, antivirus y antispam ¡gratis!
¡Abrí tu cuenta ya! - http://correo.yahoo.com.ar
 
D

Dennis Lee Bieber

I am using os.system. Here is my code

import ftplib, posixpath, threading
from TaskQueue import TaskQueue

def worker(tq):
while True:
host, e = tq.get()

c = ftplib.FTP(host)
c.connect()
try:
c.login()
p = posixpath.basename(e)

I believe the recommendation is to use os.path; the os module
determines which variant is best suited for the host.
ps_dir = r'H:/ftp_download/'
filename = download_dir+p

So where is "download_dir" defined? And for local files at least,
use os.path.join() to clean-up all these filename/path manipulations.
fp = open(filename, 'wb')
try: c.retrbinary('RETR %s' % e, fp.write)
finally: fp.close()
finally: c.close()
if (p.lower().endswith('.ps') ):

Note that if the FTP fetch failed you are still attempting to
convert it.
partFileName = p.split('.', 1)

Use os.path.splitext()
movedFile = download_dir + p
os.path.join()
#movedFile = p
finalFile = ps_dir + partFileName[0]+'.pdf'
os.path.join()

encode_cmd = r'ps2pdf '+ movedFile + ' '+ finalFile
os.system(encode_cmd)

Might be cleaner to use
os.system("ps2pdf %s %s" % (movedFile, finalFile) )
tq.task_done()

if __name__ == '__main__':
main()

def main():
q = TaskQueue()
#host = 'ftp.microsoft.com'
host = 'mysite.com'
c = ftplib.FTP(host)
c.connect()
try:
#c.login()
c.login()

#folder = '/deskapps/kids/'
folder = ''
for n in c.nlst(folder):
if n.lower().endswith('.ps'):
q.put((host, n))
I am giving unique name to the converted file with .pdf. I think
something up with the thread. I am new to python, I am not sure what's
wrong.

"new to python", and you don't consider using threading, ftplib,
etc. rather advanced to start with?

I'd have thought getting the processing of ONE file working reliably
to be the first concern -- then plug that processing into a threading
environment.

--
Wulfraed Dennis Lee Bieber KD6MOG
(e-mail address removed) (e-mail address removed)
HTTP://wlfraed.home.netcom.com/
(Bestiaria Support Staff: (e-mail address removed))
HTTP://www.bestiaria.com/
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,734
Messages
2,569,441
Members
44,832
Latest member
GlennSmall

Latest Threads

Top