mysql bulk insert from python

[Dave Ekhaus]

hi

my news carrier didn't have any mysql specific news groups - so
apologize in advance if this isn't exactly the proper group for these
question(s) ...

anyway - i'm trying to figure out how to perform a bulk insert of data
from a text file into a table in a mysql database from the python
programming language. i have lots of experience with ms sql server and
there we used bcp and then dts packages, but i can't seem to find the
analogous method for mysql - and would appreciate any tips/insight the
group can offer.

fwiw - the data files contain on the order of 100,000 + records
(sometimes approx. 1 million), so anything but a 'bulk' solution
probably isn't practical.

also - i found some reference to a 'Loader' python module, but it
seems to related to MaxDB and SAP. i'm not sure what that is - and
would really prefer a more generic solution.

thanks
dave

 

[Wolfram Kraus]

hi

my news carrier didn't have any mysql specific news groups - so
apologize in advance if this isn't exactly the proper group for these
question(s) ...

anyway - i'm trying to figure out how to perform a bulk insert of
data from a text file into a table in a mysql database from the python
programming language. i have lots of experience with ms sql server and
there we used bcp and then dts packages, but i can't seem to find the
analogous method for mysql - and would appreciate any tips/insight the
group can offer.

fwiw - the data files contain on the order of 100,000 + records
(sometimes approx. 1 million), so anything but a 'bulk' solution
probably isn't practical.

also - i found some reference to a 'Loader' python module, but it
seems to related to MaxDB and SAP. i'm not sure what that is - and
would really prefer a more generic solution.

thanks
dave

The MySQL command LOAD DATA INFILE
(http://dev.mysql.com/doc/mysql/en/LOAD_DATA.html) might be what you are
looking for, no need for python here, but you can call this command from
any python script that uses the MySQLDB module
(http://sourceforge.net/projects/mysql-python)


HTH,
Wolfram

PS: Maybe news.gmane.org has some mysql-lists that you can try