mySql and multiple connection for threads

[johnny]

How do you create multiple connection in the treads. Lets say I will
have at most 5 threads and I want to create at most 5 connections. If
I create a connection in the "worker method", does it create connection
for each threads.

def worker(tq):
while True:
host, e = tq.get()

c = ftplib.FTP(host)
c.connect()
try:
c.login()
p = os.path.basename(e)
download_dir = r'H:/ftp_download/'
ps_dir = r'H:/ftp_download/'
filename = download_dir+p
fp = open(filename, 'wb')
try: c.retrbinary('RETR %s' % e, fp.write)
finally: fp.close()
finally: c.close()
if (p.lower().endswith('.ps') ):
partFileName = p.split('.', 1)
movedFile = download_dir + p
#movedFile = p
finalFile = ps_dir + partFileName[0]+'.pdf'
encode_cmd = r'ps2pdf '+ movedFile + ' '+ finalFile
os.system(encode_cmd)

conn = adodb.NewADOConnection('mysql')
conn.Connect('localhost', 'temp', 'temp', 'temp')
sql = r"update file where file_path='"+p+"' set
pdf_file_path='" +finalFile+"'"
cursor = conn.Execute(sql)
rows = cursor.Affected_Rows()

tq.task_done()

 

[Dennis Lee Bieber]

How do you create multiple connection in the treads. Lets say I will
have at most 5 threads and I want to create at most 5 connections. If
I create a connection in the "worker method", does it create connection
for each threads.

It should... Each thread has its own stack, and "variables" local to
a function are stored in the thread-specific stack...

def worker(tq):
while True:
host, e = tq.get()

c = ftplib.FTP(host)
c.connect()
try:
c.login()
p = os.path.basename(e)
download_dir = r'H:/ftp_download/'

Ah, this time you have defined a value for "download_dir"... But I
notice it is identical to "ps_dir"... Sounds like a maintenance
nightmare -- two items that need to be changed to implement what is
logically just one change...

ps_dir = r'H:/ftp_download/'

Somewhat more confusingly... "download_dir" is where you are
"moving" the Postscript files /to/ in the FTP process... And "ps_dir" is
where you are creating PDF files...

Note: since you are using forward / you don't need the safety margin
of using raw strings.

filename = download_dir+p

Please consider using the os.path.* functions for manipulation of
file/path names. If, for example, the trailing / were missing from
"download_dir", the above will fail.

fp = open(filename, 'wb')
try: c.retrbinary('RETR %s' % e, fp.write)
finally: fp.close()
finally: c.close()
if (p.lower().endswith('.ps') ):

if the c.login() fails, there will not be a "p", this will raise an
exception.

partFileName = p.split('.', 1)

The preceding "if" has already confirmed the name ends with the
three characters ".ps"... If all you need is the part in front of it you
could just do

partFileName = p[:-3]

p[:-3]

'H:\\ftp_download\\junk'

However, given that "p" might not exist...

try:
(root, ext) = os.path.splitext(p)
if ext.lower() == ".ps":

encode_cmd = "ps2pdf %s %s" % (
os.join(download_dir, p),
os.join(ps_dir, root + ".pdf") )
#unfortunately, join() only works on directories, isn't smart enough
#to join an extension

Note that, if download_dir or ps_dir have spaces in the path(s), ...

movedFile = download_dir + p
#movedFile = p
finalFile = ps_dir + partFileName[0]+'.pdf'
encode_cmd = r'ps2pdf '+ movedFile + ' '+ finalFile
os.system(encode_cmd)

... this will fail.

Might be better to use

encode_cmd = 'ps2pdf "%s" "%s" ' % ( ...

so each file/path is surrounded by "s

conn = adodb.NewADOConnection('mysql')

I don't do adodb... you'd have to check M$ documentation regarding
thread safety and multiple connections. I'm presuming it is using an
ODBC driver with an ODBC datasource pre-defined.

Is adodb even DB-API compliant?

Personally, I'd install a version of MySQLdb that is compatible with
the server and the Python version (one reason I'm still on Python 2.4.x
-- not all of the adapters I use have been built into stand-alone
Windows installers for 2.5 yet; and I don't have a native build
facility).

MySQLdb has a thread-safety of 1, meaning each thread has to make
its own connection.

Using the os.path.* and a DB-API compliant module would assist in
making the code less system dependent.

conn.Connect('localhost', 'temp', 'temp', 'temp')

conn = MySQLdb.connect(host="localhost", user="temp",
passwd="temp", db="temp")
curs = conn.cursor()

sql = r"update file where file_path='"+p+"' set
pdf_file_path='" +finalFile+"'"

DON'T DO THAT! Besides, the SQL syntax is invalid!

cursor = conn.Execute(sql)

rows = curs.execute("""update file
set pdf_file_path = %s
where file_path = %s""",
(finalFile, p) )

MySQLdb uses %s for parameters, other DB-API modules use ?

rows = cursor.Affected_Rows()

finally: #not the best error handling, but for completeness
#with the try I put in
curs.close()
conn.commit()
conn.close()

tq.task_done()

Not doing anything with "rows"?

--
Wulfraed Dennis Lee Bieber KD6MOG
(e-mail address removed) (e-mail address removed)
HTTP://wlfraed.home.netcom.com/
(Bestiaria Support Staff: (e-mail address removed))
HTTP://www.bestiaria.com/

 

[Dennis Lee Bieber]

This is a resend -- I suspect a hokey server (only some 20 messages
in c.l.py in 24 hours? And I'm not even seeing my own recently sent
messages!)

How do you create multiple connection in the treads. Lets say I will
have at most 5 threads and I want to create at most 5 connections. If
I create a connection in the "worker method", does it create connection
for each threads.

It should... Each thread has its own stack, and "variables" local to
a function are stored in the thread-specific stack...

def worker(tq):
while True:
host, e = tq.get()

c = ftplib.FTP(host)
c.connect()
try:
c.login()
p = os.path.basename(e)
download_dir = r'H:/ftp_download/'

Ah, this time you have defined a value for "download_dir"... But I
notice it is identical to "ps_dir"... Sounds like a maintenance
nightmare -- two items that need to be changed to implement what is
logically just one change...

ps_dir = r'H:/ftp_download/'

Somewhat more confusingly... "download_dir" is where you are
"moving" the Postscript files /to/ in the FTP process... And "ps_dir" is
where you are creating PDF files...

Note: since you are using forward / you don't need the safety margin
of using raw strings.

filename = download_dir+p

Please consider using the os.path.* functions for manipulation of
file/path names. If, for example, the trailing / were missing from
"download_dir", the above will fail.

fp = open(filename, 'wb')
try: c.retrbinary('RETR %s' % e, fp.write)
finally: fp.close()
finally: c.close()
if (p.lower().endswith('.ps') ):

if the c.login() fails, there will not be a "p", this will raise an
exception.

partFileName = p.split('.', 1)

The preceding "if" has already confirmed the name ends with the
three characters ".ps"... If all you need is the part in front of it you
could just do

partFileName = p[:-3]

p[:-3]

'H:\\ftp_download\\junk'

However, given that "p" might not exist...

try:
(root, ext) = os.path.splitext(p)
if ext.lower() == ".ps":

encode_cmd = "ps2pdf %s %s" % (
os.join(download_dir, p),
os.join(ps_dir, root + ".pdf") )
#unfortunately, join() only works on directories, isn't smart enough
#to join an extension

Note that, if download_dir or ps_dir have spaces in the path(s), ...

movedFile = download_dir + p
#movedFile = p
finalFile = ps_dir + partFileName[0]+'.pdf'
encode_cmd = r'ps2pdf '+ movedFile + ' '+ finalFile
os.system(encode_cmd)

... this will fail.

Might be better to use

encode_cmd = 'ps2pdf "%s" "%s" ' % ( ...

so each file/path is surrounded by "s

conn = adodb.NewADOConnection('mysql')

I don't do adodb... you'd have to check M$ documentation regarding
thread safety and multiple connections. I'm presuming it is using an
ODBC driver with an ODBC datasource pre-defined.

Is adodb even DB-API compliant?

Personally, I'd install a version of MySQLdb that is compatible with
the server and the Python version (one reason I'm still on Python 2.4.x
-- not all of the adapters I use have been built into stand-alone
Windows installers for 2.5 yet; and I don't have a native build
facility).

MySQLdb has a thread-safety of 1, meaning each thread has to make
its own connection.

Using the os.path.* and a DB-API compliant module would assist in
making the code less system dependent.

conn.Connect('localhost', 'temp', 'temp', 'temp')

conn = MySQLdb.connect(host="localhost", user="temp",
passwd="temp", db="temp")
curs = conn.cursor()

sql = r"update file where file_path='"+p+"' set
pdf_file_path='" +finalFile+"'"

DON'T DO THAT! Besides, the SQL syntax is invalid!

cursor = conn.Execute(sql)

rows = curs.execute("""update file
set pdf_file_path = %s
where file_path = %s""",
(finalFile, p) )

MySQLdb uses %s for parameters, other DB-API modules use ?

rows = cursor.Affected_Rows()

finally: #not the best error handling, but for completeness
#with the try I put in
curs.close()
conn.commit()
conn.close()

tq.task_done()

Not doing anything with "rows"?

--
Wulfraed Dennis Lee Bieber KD6MOG
(e-mail address removed) (e-mail address removed)
HTTP://wlfraed.home.netcom.com/
(Bestiaria Support Staff: (e-mail address removed))
HTTP://www.bestiaria.com/

 

[johnny]

Thank you VERY MUCH Dennis, for taking the time to explain to me.

 

[johnny]

Another question I have is, let say you get an exception and I want to
write all exception into a log file. There is one log file, and 5
threads will be sharing. I need a way to lock it, write the log, and
release the log file, so other threads can write to the logs. Can some
one give me some pointers?

 

[Dennis Lee Bieber]

Another question I have is, let say you get an exception and I want to
write all exception into a log file. There is one log file, and 5
threads will be sharing. I need a way to lock it, write the log, and
release the log file, so other threads can write to the logs. Can some
one give me some pointers?

import logging

and then look at whatever documentation is available for the module...

resuming the actual logging module followed the PEP, which is quoted
herewith:

-=-=-=-=-=-=-
Thread Safety
The logging system should support thread-safe operation without
any special action needing to be taken by its users.
-=-=-=-=-=-=-

And since I just found...

-=-=-=-=-=-=-
6.29.9 Thread Safety

The logging module is intended to be thread-safe without any special
work needing to be done by its clients. It achieves this though using
threading locks; there is one lock to serialize access to the module's
shared data, and each handler also creates a lock to serialize access to
its underlying I/O.
-=-=-=-=-=-=-

There isn't much you need to do, other than import the module, and
have each thread (just so you can track which one logged what) get
loggers of the form "AppName.ThreadName". Each thread will use "its"
logger, and if the only handler is really the master logger configured
before starting threads, everything should go out to one log file...

--
Wulfraed Dennis Lee Bieber KD6MOG
(e-mail address removed) (e-mail address removed)
HTTP://wlfraed.home.netcom.com/
(Bestiaria Support Staff: (e-mail address removed))
HTTP://www.bestiaria.com/