mySql and multiple connection for threads

Discussion in 'Python' started by johnny, Dec 8, 2006.

  1. johnny

    johnny Guest

    How do you create multiple connection in the treads. Lets say I will
    have at most 5 threads and I want to create at most 5 connections. If
    I create a connection in the "worker method", does it create connection
    for each threads.

    def worker(tq):
    while True:
    host, e = tq.get()

    c = ftplib.FTP(host)
    c.connect()
    try:
    c.login()
    p = os.path.basename(e)
    download_dir = r'H:/ftp_download/'
    ps_dir = r'H:/ftp_download/'
    filename = download_dir+p
    fp = open(filename, 'wb')
    try: c.retrbinary('RETR %s' % e, fp.write)
    finally: fp.close()
    finally: c.close()
    if (p.lower().endswith('.ps') ):
    partFileName = p.split('.', 1)
    movedFile = download_dir + p
    #movedFile = p
    finalFile = ps_dir + partFileName[0]+'.pdf'
    encode_cmd = r'ps2pdf '+ movedFile + ' '+ finalFile
    os.system(encode_cmd)

    conn = adodb.NewADOConnection('mysql')
    conn.Connect('localhost', 'temp', 'temp', 'temp')
    sql = r"update file where file_path='"+p+"' set
    pdf_file_path='" +finalFile+"'"
    cursor = conn.Execute(sql)
    rows = cursor.Affected_Rows()

    tq.task_done()
    johnny, Dec 8, 2006
    #1
    1. Advertising

  2. On 8 Dec 2006 10:23:09 -0800, "johnny" <> declaimed
    the following in comp.lang.python:

    > How do you create multiple connection in the treads. Lets say I will
    > have at most 5 threads and I want to create at most 5 connections. If
    > I create a connection in the "worker method", does it create connection
    > for each threads.
    >

    It should... Each thread has its own stack, and "variables" local to
    a function are stored in the thread-specific stack...

    > def worker(tq):
    > while True:
    > host, e = tq.get()
    >
    > c = ftplib.FTP(host)
    > c.connect()
    > try:
    > c.login()
    > p = os.path.basename(e)
    > download_dir = r'H:/ftp_download/'


    Ah, this time you have defined a value for "download_dir"... But I
    notice it is identical to "ps_dir"... Sounds like a maintenance
    nightmare -- two items that need to be changed to implement what is
    logically just one change...

    > ps_dir = r'H:/ftp_download/'


    Somewhat more confusingly... "download_dir" is where you are
    "moving" the Postscript files /to/ in the FTP process... And "ps_dir" is
    where you are creating PDF files...

    Note: since you are using forward / you don't need the safety margin
    of using raw strings.

    > filename = download_dir+p


    Please consider using the os.path.* functions for manipulation of
    file/path names. If, for example, the trailing / were missing from
    "download_dir", the above will fail.

    >>> import os.path
    >>> down_dir = "H:\\ftp_download\\"
    >>> p = "junk.ps" #for test
    >>>
    >>> os.path.join(down_dir, p)

    'H:\\ftp_download\\junk.ps'
    >>> down_dir + p

    'H:\\ftp_download\\junk.ps'
    >>> down_dir = "H:\\ftp_download" #no slash
    >>> os.path.join(down_dir, p)

    'H:\\ftp_download\\junk.ps'
    >>> down_dir + p

    'H:\\ftp_downloadjunk.ps'
    >>>


    > fp = open(filename, 'wb')
    > try: c.retrbinary('RETR %s' % e, fp.write)
    > finally: fp.close()
    > finally: c.close()
    > if (p.lower().endswith('.ps') ):


    if the c.login() fails, there will not be a "p", this will raise an
    exception.

    > partFileName = p.split('.', 1)


    The preceding "if" has already confirmed the name ends with the
    three characters ".ps"... If all you need is the part in front of it you
    could just do

    partFileName = p[:-3]

    >>> p[:-3]

    'H:\\ftp_download\\junk'

    However, given that "p" might not exist...

    try:
    (root, ext) = os.path.splitext(p)
    if ext.lower() == ".ps":

    encode_cmd = "ps2pdf %s %s" % (
    os.join(download_dir, p),
    os.join(ps_dir, root + ".pdf") )
    #unfortunately, join() only works on directories, isn't smart enough
    #to join an extension

    Note that, if download_dir or ps_dir have spaces in the path(s), ...

    > movedFile = download_dir + p
    > #movedFile = p
    > finalFile = ps_dir + partFileName[0]+'.pdf'
    > encode_cmd = r'ps2pdf '+ movedFile + ' '+ finalFile
    > os.system(encode_cmd)


    ... this will fail.

    Might be better to use

    encode_cmd = 'ps2pdf "%s" "%s" ' % ( ...

    so each file/path is surrounded by "s
    >
    > conn = adodb.NewADOConnection('mysql')


    I don't do adodb... you'd have to check M$ documentation regarding
    thread safety and multiple connections. I'm presuming it is using an
    ODBC driver with an ODBC datasource pre-defined.

    Is adodb even DB-API compliant?

    Personally, I'd install a version of MySQLdb that is compatible with
    the server and the Python version (one reason I'm still on Python 2.4.x
    -- not all of the adapters I use have been built into stand-alone
    Windows installers for 2.5 yet; and I don't have a native build
    facility).

    MySQLdb has a thread-safety of 1, meaning each thread has to make
    its own connection.

    Using the os.path.* and a DB-API compliant module would assist in
    making the code less system dependent.

    > conn.Connect('localhost', 'temp', 'temp', 'temp')


    conn = MySQLdb.connect(host="localhost", user="temp",
    passwd="temp", db="temp")
    curs = conn.cursor()


    > sql = r"update file where file_path='"+p+"' set
    > pdf_file_path='" +finalFile+"'"


    DON'T DO THAT! Besides, the SQL syntax is invalid!

    > cursor = conn.Execute(sql)


    rows = curs.execute("""update file
    set pdf_file_path = %s
    where file_path = %s""",
    (finalFile, p) )

    MySQLdb uses %s for parameters, other DB-API modules use ?

    > rows = cursor.Affected_Rows()
    >

    finally: #not the best error handling, but for completeness
    #with the try I put in
    curs.close()
    conn.commit()
    conn.close()

    > tq.task_done()


    Not doing anything with "rows"?

    --
    Wulfraed Dennis Lee Bieber KD6MOG

    HTTP://wlfraed.home.netcom.com/
    (Bestiaria Support Staff: )
    HTTP://www.bestiaria.com/
    Dennis Lee Bieber, Dec 9, 2006
    #2
    1. Advertising

  3. This is a resend -- I suspect a hokey server (only some 20 messages
    in c.l.py in 24 hours? And I'm not even seeing my own recently sent
    messages!)

    On 8 Dec 2006 10:23:09 -0800, "johnny" <> declaimed
    the following in comp.lang.python:

    > How do you create multiple connection in the treads. Lets say I will
    > have at most 5 threads and I want to create at most 5 connections. If
    > I create a connection in the "worker method", does it create connection
    > for each threads.
    >

    It should... Each thread has its own stack, and "variables" local to
    a function are stored in the thread-specific stack...

    > def worker(tq):
    > while True:
    > host, e = tq.get()
    >
    > c = ftplib.FTP(host)
    > c.connect()
    > try:
    > c.login()
    > p = os.path.basename(e)
    > download_dir = r'H:/ftp_download/'


    Ah, this time you have defined a value for "download_dir"... But I
    notice it is identical to "ps_dir"... Sounds like a maintenance
    nightmare -- two items that need to be changed to implement what is
    logically just one change...

    > ps_dir = r'H:/ftp_download/'


    Somewhat more confusingly... "download_dir" is where you are
    "moving" the Postscript files /to/ in the FTP process... And "ps_dir" is
    where you are creating PDF files...

    Note: since you are using forward / you don't need the safety margin
    of using raw strings.

    > filename = download_dir+p


    Please consider using the os.path.* functions for manipulation of
    file/path names. If, for example, the trailing / were missing from
    "download_dir", the above will fail.

    >>> import os.path
    >>> down_dir = "H:\\ftp_download\\"
    >>> p = "junk.ps" #for test
    >>>
    >>> os.path.join(down_dir, p)

    'H:\\ftp_download\\junk.ps'
    >>> down_dir + p

    'H:\\ftp_download\\junk.ps'
    >>> down_dir = "H:\\ftp_download" #no slash
    >>> os.path.join(down_dir, p)

    'H:\\ftp_download\\junk.ps'
    >>> down_dir + p

    'H:\\ftp_downloadjunk.ps'
    >>>


    > fp = open(filename, 'wb')
    > try: c.retrbinary('RETR %s' % e, fp.write)
    > finally: fp.close()
    > finally: c.close()
    > if (p.lower().endswith('.ps') ):


    if the c.login() fails, there will not be a "p", this will raise an
    exception.

    > partFileName = p.split('.', 1)


    The preceding "if" has already confirmed the name ends with the
    three characters ".ps"... If all you need is the part in front of it you
    could just do

    partFileName = p[:-3]

    >>> p[:-3]

    'H:\\ftp_download\\junk'

    However, given that "p" might not exist...

    try:
    (root, ext) = os.path.splitext(p)
    if ext.lower() == ".ps":

    encode_cmd = "ps2pdf %s %s" % (
    os.join(download_dir, p),
    os.join(ps_dir, root + ".pdf") )
    #unfortunately, join() only works on directories, isn't smart enough
    #to join an extension

    Note that, if download_dir or ps_dir have spaces in the path(s), ...

    > movedFile = download_dir + p
    > #movedFile = p
    > finalFile = ps_dir + partFileName[0]+'.pdf'
    > encode_cmd = r'ps2pdf '+ movedFile + ' '+ finalFile
    > os.system(encode_cmd)


    ... this will fail.

    Might be better to use

    encode_cmd = 'ps2pdf "%s" "%s" ' % ( ...

    so each file/path is surrounded by "s
    >
    > conn = adodb.NewADOConnection('mysql')


    I don't do adodb... you'd have to check M$ documentation regarding
    thread safety and multiple connections. I'm presuming it is using an
    ODBC driver with an ODBC datasource pre-defined.

    Is adodb even DB-API compliant?

    Personally, I'd install a version of MySQLdb that is compatible with
    the server and the Python version (one reason I'm still on Python 2.4.x
    -- not all of the adapters I use have been built into stand-alone
    Windows installers for 2.5 yet; and I don't have a native build
    facility).

    MySQLdb has a thread-safety of 1, meaning each thread has to make
    its own connection.

    Using the os.path.* and a DB-API compliant module would assist in
    making the code less system dependent.

    > conn.Connect('localhost', 'temp', 'temp', 'temp')


    conn = MySQLdb.connect(host="localhost", user="temp",
    passwd="temp", db="temp")
    curs = conn.cursor()


    > sql = r"update file where file_path='"+p+"' set
    > pdf_file_path='" +finalFile+"'"


    DON'T DO THAT! Besides, the SQL syntax is invalid!

    > cursor = conn.Execute(sql)


    rows = curs.execute("""update file
    set pdf_file_path = %s
    where file_path = %s""",
    (finalFile, p) )

    MySQLdb uses %s for parameters, other DB-API modules use ?

    > rows = cursor.Affected_Rows()
    >

    finally: #not the best error handling, but for completeness
    #with the try I put in
    curs.close()
    conn.commit()
    conn.close()

    > tq.task_done()


    Not doing anything with "rows"?

    --
    Wulfraed Dennis Lee Bieber KD6MOG

    HTTP://wlfraed.home.netcom.com/
    (Bestiaria Support Staff: )
    HTTP://www.bestiaria.com/
    Dennis Lee Bieber, Dec 9, 2006
    #3
  4. johnny

    johnny Guest

    Thank you VERY MUCH Dennis, for taking the time to explain to me.
    johnny, Dec 10, 2006
    #4
  5. johnny

    johnny Guest

    Another question I have is, let say you get an exception and I want to
    write all exception into a log file. There is one log file, and 5
    threads will be sharing. I need a way to lock it, write the log, and
    release the log file, so other threads can write to the logs. Can some
    one give me some pointers?
    johnny, Dec 10, 2006
    #5
  6. On 10 Dec 2006 13:15:14 -0800, "johnny" <> declaimed
    the following in gmane.comp.python.general:

    > Another question I have is, let say you get an exception and I want to
    > write all exception into a log file. There is one log file, and 5
    > threads will be sharing. I need a way to lock it, write the log, and
    > release the log file, so other threads can write to the logs. Can some
    > one give me some pointers?


    import logging

    and then look at whatever documentation is available for the module...

    resuming the actual logging module followed the PEP, which is quoted
    herewith:

    -=-=-=-=-=-=-
    Thread Safety
    The logging system should support thread-safe operation without
    any special action needing to be taken by its users.
    -=-=-=-=-=-=-

    And since I just found...

    -=-=-=-=-=-=-
    6.29.9 Thread Safety

    The logging module is intended to be thread-safe without any special
    work needing to be done by its clients. It achieves this though using
    threading locks; there is one lock to serialize access to the module's
    shared data, and each handler also creates a lock to serialize access to
    its underlying I/O.
    -=-=-=-=-=-=-

    There isn't much you need to do, other than import the module, and
    have each thread (just so you can track which one logged what) get
    loggers of the form "AppName.ThreadName". Each thread will use "its"
    logger, and if the only handler is really the master logger configured
    before starting threads, everything should go out to one log file...

    --
    Wulfraed Dennis Lee Bieber KD6MOG

    HTTP://wlfraed.home.netcom.com/
    (Bestiaria Support Staff: )
    HTTP://www.bestiaria.com/
    Dennis Lee Bieber, Dec 11, 2006
    #6
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Smegly
    Replies:
    1
    Views:
    1,096
    Mitchell
    May 19, 2004
  2. johnny
    Replies:
    1
    Views:
    791
    Jerry Hill
    Dec 8, 2006
  3. Jeffrey H. Coffield
    Replies:
    1
    Views:
    1,853
  4. Une bévue
    Replies:
    0
    Views:
    140
    Une bévue
    Jun 14, 2006
  5. Marc Weber
    Replies:
    1
    Views:
    147
    Marc Weber
    Feb 26, 2010
Loading...

Share This Page