Include pysqlite2 into Python 2.5?

G

Gerhard Haering

[This is a repost from a message to python-dev on the users' list,
because python-dev-ers thought a broader discussion of this is a good
idea.]

Last December, we had a short thread discussing (in python-dev) the
integration of PySQLite into Python 2.4. At the time, I was against
inclusion, because I thought PySQLite was not ripe for it, mostly
because I thought the API was not stable.

Now, I have started writing a new PySQLite module, which has the
following key features:

- Uses iterator-style SQLite 3.x API: sqlite3_compile, sqlite3_step()
etc. This way, it is possible to use prepared statements, and for
large resultsets, it requires less memory, because the whole
resultset isn't fetched into memory at once any longer.

- Completely incompatible with the SQLite 0.x/1.x API: I'm free to
create a much better API now.

- "In the face of ambiguity, refuse the temptation to guess." -
PySQLite 1.x tries to "guess" which Python type to convert to. It's
pretty good at it, because it queries the column type information.
This works for, I'd say 90 % of all cases at least. But as soon as
you use anything fancy like functions, aggregates or expressions in
SQL, the _typeless_ nature of SQLite breaks through and it will tell
us nothing about the declared column type (of course, because the
data is not coming from a database column).

So I decided to change the default behaviour and make PySQLite
typeless by default, too. Everything will be returned as a Unicode
string (the default might be user-configurable per connection).

Unless, unless of course the user explicitly activates the
"guess-mode" ;-) But to do so, she must read the docs then she will
be aware of the fact that it only works in 90 % of all cases.

So why am I bothering you about this?

I think that a simple embedded relational database would be a good
thing to have in Python by default. And as Python 2.5 won't happen
anytime soon, there's plenty of time for developing it, getting it
stable, and integrating it.

Especially those of you that have used PySQLite in the past, do you
have any suggestions that would make the rewrite a better candidate
for inclusion into Python?

One problem I see is that even the new PySQLite will grow and try to
wrap much of the SQLite API that are not directly related to the
DB-API. If such a thing is too complicated/big for the standard
library, then maybe it would be better to produce a much simpler
PySQLite, especially for the Python standard library that leaves all
the fancy stuff out. My codename would be "embsql".

So, what would you like to see? "import sqlite", "import embsql", or
"pypi.install('pysqlite')" ?

-- Gerhard

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFBd2/EdIO4ozGCH14RAhKpAJ9F33gxhA2j+y/CeuKe0wvFpRZAKwCePMEL
dzvLQjiR6GaRS8M3eq2GXYY=
=SEKQ
-----END PGP SIGNATURE-----
 
G

Ganesan R

Gerhard" == Gerhard Haering said:
Last December, we had a short thread discussing (in python-dev) the
integration of PySQLite into Python 2.4. At the time, I was against
inclusion, because I thought PySQLite was not ripe for it, mostly
because I thought the API was not stable.

Thank you for PySQLite and congratulations for reaching the 1.0 milestone. I
have only experimented with PySQLite so far but it's definitely in my
toolbox for any future projects :).
I think that a simple embedded relational database would be a good
thing to have in Python by default. And as Python 2.5 won't happen
anytime soon, there's plenty of time for developing it, getting it
stable, and integrating it.

+1. bsddb is in there and SQLite definitely deserves to be included. A free
license, no setup, easy to use - what more can you ask for :).
Especially those of you that have used PySQLite in the past, do you
have any suggestions that would make the rewrite a better candidate
for inclusion into Python?

I have not used it extensively to be able to get meaningful suggestions. My
personal peeve is not with the PySQLite API but with DB-API 2.0. As some one
(Alex Martelli?) mentioned in another thread, it's high time Python
standardized on one (may be two) mandatory param styles.
One problem I see is that even the new PySQLite will grow and try to
wrap much of the SQLite API that are not directly related to the
DB-API. If such a thing is too complicated/big for the standard
library, then maybe it would be better to produce a much simpler
PySQLite, especially for the Python standard library that leaves all
the fancy stuff out. My codename would be "embsql".
So, what would you like to see? "import sqlite", "import embsql", or
"pypi.install('pysqlite')" ?

import sqlite

Ganesan
 
I

Istvan Albert

Gerhard said:
Especially those of you that have used PySQLite in the past, do you
have any suggestions that would make the rewrite a better candidate
for inclusion into Python?

great idea!

I'm currently in the planning phase of moving an application
from a relational database to SQLite via PySQL. I think
embedded, SQL capable databases are "the next big thing"
there is just to much pain being felt by having to manage a
big database when you don't really need them ....
One problem I see is that even the new PySQLite will grow and try to
wrap much of the SQLite API that are not directly related to the
DB-API. If such a thing is too complicated/big for the standard
library, then maybe it would be better to produce a much simpler
PySQLite, especially for the Python standard library that leaves all
the fancy stuff out. My codename would be "embsql".

So, what would you like to see? "import sqlite", "import embsql", or

I think import sqlite would be the sensible thing. Just as you said ...
.... refuse the temptation to guess ...

Istvan.
 
J

John Fabiani

Great idea and about time. Python needs some type of default SQL engine. I
have not reviewed what you are doing but may I suggest you take a look at
what VFP did with it's built-in data engine. VFP uses the concept of a
cursor along with many built-in commands/functions/statements to work with
the data.

My only question is what happens when SQLLite becomes out of favor i.e. TK
is today?

John
 
A

Alex Martelli

John Fabiani said:
Great idea and about time. Python needs some type of default SQL engine. I
have not reviewed what you are doing but may I suggest you take a look at
what VFP did with it's built-in data engine. VFP uses the concept of a
cursor along with many built-in commands/functions/statements to work with
the data.

My only question is what happens when SQLLite becomes out of favor i.e. TK
is today?

Same as what happens when BSDDB "becomes out of favor", or SAX, etc,
etc. Python either keeps supporting it (likely), moves it to a separate
download rather than bundling it (sensible), or drops it (unlikely but
not impossible) at some major release. We're not talking about stuff
that's particularly "fickle": Tkinter did its job excellently well for
MANY years, after all, to take your same example.


Alex
 
B

Bengt Richter

Same as what happens when BSDDB "becomes out of favor", or SAX, etc,
etc. Python either keeps supporting it (likely), moves it to a separate
download rather than bundling it (sensible), or drops it (unlikely but
not impossible) at some major release. We're not talking about stuff
that's particularly "fickle": Tkinter did its job excellently well for
MANY years, after all, to take your same example.
ISTM there could be a useful compromise between inclusion and separate downloads.
I.e., useful addons could be included in the core distribution as proxy stubs
in an add-on package, so that you could interactively type e.g.,

from addons import sqlite as sql

and if you had not previously done that, the stub would ask you if you'd
like to download and install the thing in question. You could cancel, or
proceed, perhaps with various options. E.g. PIL might let you exclude jpg
support if you didn't want it, and so forth. IIRC&UIAM the BSD ports system
has this kind of functionality -- i.e., instead of full packages, there's
automated download (or off CD) install info pre-packaged.

Options just to download and view release notes or to download docs only
for looking over before installing all might be useful sometimes also.

Like wise, there could be an option to install under addons or site-packages.
And I'm sure people would think of other handy stuff.

There would have to be security checks with md5's or pgp signatures etc.,
but IWT the aproach could be made to work. Official addons could live be
anywhere, even with multiple mirrors, and different components used by
higher level packages could have different urls, but the location urls, sizes, and
hashes would be registered at python.org, so a stub would go there to find
addon info for a particular version of python. This would also guarantee
compatibility (or a notice that no compatible set of required parts is available).

Just a thought.

Regards,
Bengt Richter
 
R

Robert

Gerhard said:
[This is a repost from a message to python-dev on the users' list,
because python-dev-ers thought a broader discussion of this is a good
idea.]

Last December, we had a short thread discussing (in python-dev) the
integration of PySQLite into Python 2.4. At the time, I was against
inclusion, because I thought PySQLite was not ripe for it, mostly
because I thought the API was not stable.

Now, I have started writing a new PySQLite module, which has the
following key features:

- Uses iterator-style SQLite 3.x API: sqlite3_compile, sqlite3_step()
etc. This way, it is possible to use prepared statements, and for
large resultsets, it requires less memory, because the whole
resultset isn't fetched into memory at once any longer.

- Completely incompatible with the SQLite 0.x/1.x API: I'm free to
create a much better API now.

- "In the face of ambiguity, refuse the temptation to guess." -
PySQLite 1.x tries to "guess" which Python type to convert to. It's
pretty good at it, because it queries the column type information.
This works for, I'd say 90 % of all cases at least. But as soon as
you use anything fancy like functions, aggregates or expressions in
SQL, the _typeless_ nature of SQLite breaks through and it will tell
us nothing about the declared column type (of course, because the
data is not coming from a database column).

So I decided to change the default behaviour and make PySQLite
typeless by default, too. Everything will be returned as a Unicode
string (the default might be user-configurable per connection).

Unless, unless of course the user explicitly activates the
"guess-mode" ;-) But to do so, she must read the docs then she will
be aware of the fact that it only works in 90 % of all cases.

So why am I bothering you about this?

I think that a simple embedded relational database would be a good
thing to have in Python by default. And as Python 2.5 won't happen
anytime soon, there's plenty of time for developing it, getting it
stable, and integrating it.

Especially those of you that have used PySQLite in the past, do you
have any suggestions that would make the rewrite a better candidate
for inclusion into Python?

One problem I see is that even the new PySQLite will grow and try to
wrap much of the SQLite API that are not directly related to the
DB-API. If such a thing is too complicated/big for the standard
library, then maybe it would be better to produce a much simpler
PySQLite, especially for the Python standard library that leaves all
the fancy stuff out. My codename would be "embsql".

So, what would you like to see? "import sqlite", "import embsql", or
"pypi.install('pysqlite')" ?

-- Gerhard
"import sqlite"

Do we really have to wait unitil Python 2.5 to get it? ;-)

Robert
 
D

Dave Cook

John Fabiani said:
Great idea and about time. Python needs some type of default SQL engine.

<AOL>
Me, too!
</AOL>

I use sqlite heavily (via SQLObject and the current pysqlite).

Dave Cook
 
G

Gerhard Haering

Do we really have to wait unitil Python 2.5 to get it? ;-)

It's too late for Python 2.4, which is in beta stage now. Also,
PySQLite 2.0, the one we're talking about here, is still pre-alpha.

-- Gerhard

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFBeMg/dIO4ozGCH14RAl7YAKC106WWYUDLxWe+3YBq3ndRm4y+owCfUtjb
R3ARGUytJ5KU0eLG3kpKf0U=
=/Uj/
-----END PGP SIGNATURE-----
 
W

Wilk

Gerhard Haering said:
So, what would you like to see? "import sqlite", "import embsql", or
"pypi.install('pysqlite')" ?

apt-get install python-sqlite ;-)

I would like to see sqlite in python, the inclusion of sqlite in php was
greatly apreciated. I often use it for prototyping.

But for new users who will learn with sqlite, they will be disapointed
when they will want to use an other database, because of different
paramstyle, different datetime object... So i think before it could be
better to improve dbapi for user application side, and make pysqlite a
reference of implementation to follow (one paramstyle, return python
datetime, fetchdict...). If not it shall not be include imho.

We have the same problem with the web frameworks and server. I would
like to see a wsgi equivalent for database. To can plug any framework on
any database.

But when we look at the db-sig ml, it seems that there is a lot of work
to achieve this... btw guido was pushing to return python datetime.
 
R

ralobao

I think Python should expand its library of modules. It needs to
become more concise, its horrible to search the web for a module...

Sorry but its my thought.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,484
Members
44,903
Latest member
orderPeak8CBDGummies

Latest Threads

Top