Re: Eliminate "extra" variable

Discussion in 'Python' started by Mark Lawrence, Dec 8, 2013.

  1. On 08/12/2013 18:58, Tim Chase wrote:
    > On 2013-12-07 23:14, Igor Korot wrote:


    [big snip]

    >


    Whenever I need date manipulations I always reach out to this
    http://labix.org/python-dateutil

    --
    My fellow Pythonistas, ask not what our language can do for you, ask
    what you can do for our language.

    Mark Lawrence
    Mark Lawrence, Dec 8, 2013
    #1
    1. Advertising

  2. Mark Lawrence

    Roy Smith Guest

    In article <>,
    Mark Lawrence <> wrote:

    > On 08/12/2013 18:58, Tim Chase wrote:
    > > On 2013-12-07 23:14, Igor Korot wrote:

    >
    > [big snip]
    >
    > >

    >
    > Whenever I need date manipulations I always reach out to this
    > http://labix.org/python-dateutil


    The problem with dateutil is it's dog slow. Sure, I use it too, when
    convenience is more important than performance, but have you ever looked
    at the code for dateutil.parser.parse()? It's worth reading, just for
    fun.
    Roy Smith, Dec 8, 2013
    #2
    1. Advertising

  3. Mark Lawrence

    Igor Korot Guest

    Fwd: Eliminate "extra" variable

    ---------- Forwarded message ----------
    From: Igor Korot <>
    Date: Sun, Dec 8, 2013 at 12:57 PM
    Subject: Re: Eliminate "extra" variable
    To: Roy Smith <>


    Hi, guys,
    Thank you for all those valuable suggestions.
    2Tim Chase:
    I guess you missed this: "My originalData comes from the database
    query" and so the checking of the data quality is a DB burden. ;-)
    As to the function: the function purpose is to process the data and
    give out the list of dates and the dictionary of date:frequency.
    It's input is the query result, so there is no looping when the
    function is called. It is called only once.

    Also, the data comes from either SQLite or mySQL and so to eliminate
    the difference between those engines dates are processed as strings
    and converted to dates for the calculation purposes only.
    Maybe I will need to refactor SQLite processing to get the dates as
    dates and not a string, but that's probably for the future. so that
    dates will be kept as the datetime type until the end of the function.
    As I wrote the dates will be used as the text for the plotting window
    axis labels and as the labels they should come out as strings, hence
    the conversion.

    Thank you.


    On Sun, Dec 8, 2013 at 12:07 PM, Roy Smith <> wrote:
    > In article <>,
    > Mark Lawrence <> wrote:
    >
    >> On 08/12/2013 18:58, Tim Chase wrote:
    >> > On 2013-12-07 23:14, Igor Korot wrote:

    >>
    >> [big snip]
    >>
    >> >

    >>
    >> Whenever I need date manipulations I always reach out to this
    >> http://labix.org/python-dateutil

    >
    > The problem with dateutil is it's dog slow. Sure, I use it too, when
    > convenience is more important than performance, but have you ever looked
    > at the code for dateutil.parser.parse()? It's worth reading, just for
    > fun.
    > --
    > https://mail.python.org/mailman/listinfo/python-list
    Igor Korot, Dec 8, 2013
    #3
  4. Mark Lawrence

    Tim Chase Guest

    On 2013-12-08 12:58, Igor Korot wrote:
    > Also, the data comes from either SQLite or mySQL and so to eliminate
    > the difference between those engines dates are processed as strings
    > and converted to dates for the calculation purposes only.
    > Maybe I will need to refactor SQLite processing to get the dates as
    > dates and not a string, but that's probably for the future. so that
    > dates will be kept as the datetime type until the end of the
    > function. As I wrote the dates will be used as the text for the
    > plotting window axis labels and as the labels they should come out
    > as strings, hence the conversion.


    Sqlite can do this automatically if you tell it to upon connecting:

    >>> import sqlite3
    >>> conn = sqlite3.connect('x.sqlite',

    .... detect_types=sqlite3.PARSE_DECLTYPES|sqlite3.PARSE_COLNAMES)
    >>> cur.execute("create table foo (s date);")

    <sqlite3.Cursor object at 0x7f7f31665570>
    >>> import datetime
    >>> today = datetime.date.today()
    >>> cur.execute("insert into foo(s) values (?)", (today,))

    <sqlite3.Cursor object at 0x7f7f31665570>
    >>> cur.execute("select * from foo")

    <sqlite3.Cursor object at 0x7f7f31665570>
    >>> r = cur.fetchone()
    >>> print r

    (datetime.date(2013, 12, 8),)


    Note that it returns a datetime.date, the same as it was defined.

    -tkc
    Tim Chase, Dec 8, 2013
    #4
  5. Mark Lawrence

    Dave Angel Guest

    Re: Fwd: Eliminate "extra" variable

    On Sun, 8 Dec 2013 12:58:18 -0800, Igor Korot <>
    wrote:
    > It's input is the query result, so there is no looping when the
    > function is called. It is called only once.


    Then why save part of the result in an instance attribute? Just
    return all of the results as a tuple.

    --
    DaveA
    Dave Angel, Dec 8, 2013
    #5
  6. Mark Lawrence

    Igor Korot Guest

    Tim,

    On Sun, Dec 8, 2013 at 2:18 PM, Tim Chase <> wrote:
    > On 2013-12-08 12:58, Igor Korot wrote:
    >> Also, the data comes from either SQLite or mySQL and so to eliminate
    >> the difference between those engines dates are processed as strings
    >> and converted to dates for the calculation purposes only.
    >> Maybe I will need to refactor SQLite processing to get the dates as
    >> dates and not a string, but that's probably for the future. so that
    >> dates will be kept as the datetime type until the end of the
    >> function. As I wrote the dates will be used as the text for the
    >> plotting window axis labels and as the labels they should come out
    >> as strings, hence the conversion.

    >
    > Sqlite can do this automatically if you tell it to upon connecting:
    >
    >>>> import sqlite3
    >>>> conn = sqlite3.connect('x.sqlite',

    > ... detect_types=sqlite3.PARSE_DECLTYPES|sqlite3.PARSE_COLNAMES)
    >>>> cur.execute("create table foo (s date);")

    > <sqlite3.Cursor object at 0x7f7f31665570>
    >>>> import datetime
    >>>> today = datetime.date.today()
    >>>> cur.execute("insert into foo(s) values (?)", (today,))

    > <sqlite3.Cursor object at 0x7f7f31665570>
    >>>> cur.execute("select * from foo")

    > <sqlite3.Cursor object at 0x7f7f31665570>
    >>>> r = cur.fetchone()
    >>>> print r

    > (datetime.date(2013, 12, 8),)


    Interesting.
    I'm using datetime rather than the date type for CREATE TABLE() command.
    And when running SELECT I still see the b'1998-08-05 23:12:12' string
    representation when running my program under debugger.

    And it is confirmed by running this:

    >>> cur.execute("CREATE TABLE foo(bar datetime);")

    <sqlite3.Cursor object at 0x00E1E6A0>
    >>> import datetime
    >>> today = datetime.date.today()
    >>> cur.execute("insert into foo(bar) values (?)", (today,))

    <sqlite3.Cursor object at 0x00E1E6A0>
    >>> cur.execute("select * from foo")

    <sqlite3.Cursor object at 0x00E1E6A0>
    >>> res = cur.fetchall()
    >>> print res

    [(u'2013-12-14',)]
    >>>


    So, I guess this is a bug in the sqlite3 python module as datetime is
    legal data type on the DB engine.

    Thank you.

    >
    >
    > Note that it returns a datetime.date, the same as it was defined.
    >
    > -tkc
    >
    >
    >
    >
    > --
    > https://mail.python.org/mailman/listinfo/python-list
    Igor Korot, Dec 15, 2013
    #6
  7. Mark Lawrence

    Tim Chase Guest

    On 2013-12-14 23:49, Igor Korot wrote:
    > Tim,
    >
    > On Sun, Dec 8, 2013 at 2:18 PM, Tim Chase wrote:
    >>>>> conn = sqlite3.connect('x.sqlite',

    >>... detect_types=sqlite3.PARSE_DECLTYPES|sqlite3.PARSE_COLNAMES)


    Your example code omitted this one crucial line. Do you specify the
    detect_types parameter to connect()?

    -tkc
    Tim Chase, Dec 15, 2013
    #7
  8. Mark Lawrence

    Tim Chase Guest

    On 2013-12-15 06:17, Tim Chase wrote:
    >>>>>> conn = sqlite3.connect('x.sqlite',
    >>>... detect_types=sqlite3.PARSE_DECLTYPES|sqlite3.PARSE_COLNAMES)

    >
    > Your example code omitted this one crucial line. Do you specify the
    > detect_types parameter to connect()?


    It's really the PARSE_DECLTYPES that is important.

    http://docs.python.org/2/library/sqlite3.html#sqlite3.PARSE_DECLTYPES

    -tkc
    Tim Chase, Dec 15, 2013
    #8
  9. On Sat, 14 Dec 2013 23:49:58 -0800, Igor Korot <>
    declaimed the following:


    >
    >So, I guess this is a bug in the sqlite3 python module as datetime is
    >legal data type on the DB engine.
    >

    Note though, SQLite3 only has four native data types: integer, float,
    character, blob. It accepts almost anything for a type declaration, and
    will map that to the closest internal format (moreover, it will accept any
    actual datatype in any actual field -- you can store strings in an integer
    field unless the string content IS a representation of an integer in which
    case it is turned to binary).

    Regardless of how Python stores a date/datetime/time (well, except for
    the floating point C date/time types), it will mostly be kept as a string
    representation within SQLite3.

    It then becomes the responsibility of the converter modules invoked by
    the parsing option to identify and format the data. I suspect date and
    datetime both parse into the same converter. I'm not sure how it behaves if
    one defined converters for date and datetime separately, as I think it uses
    substring matching to invoke the converter: a type of date and a type of
    datetime would both match a converter for "date" if the date converter is
    first in the lookup list.


    --
    Wulfraed Dennis Lee Bieber AF6VN
    HTTP://wlfraed.home.netcom.com/
    Dennis Lee Bieber, Dec 15, 2013
    #9
  10. Mark Lawrence

    Igor Korot Guest

    Tim,

    On Sun, Dec 15, 2013 at 4:29 AM, Tim Chase
    <> wrote:
    > On 2013-12-15 06:17, Tim Chase wrote:
    >>>>>>> conn = sqlite3.connect('x.sqlite',
    >>>>... detect_types=sqlite3.PARSE_DECLTYPES|sqlite3.PARSE_COLNAMES)

    >>
    >> Your example code omitted this one crucial line. Do you specify the
    >> detect_types parameter to connect()?


    Yes, I did.
    This is the beginning of the session:

    >>> import sqlite3
    >>> conn = sqlite3.connect('c:\Documents and Settings\Igor.FORDANWORK\Desktop\mydb.db', detect_types = sqlite3.PARSE_DECLTYPES|sqlite3.PARSE_COLNAMES)


    Also please note that you session was missing the cursor creation command. ;-)

    Thank you.

    >
    > It's really the PARSE_DECLTYPES that is important.
    >
    > http://docs.python.org/2/library/sqlite3.html#sqlite3.PARSE_DECLTYPES
    >
    > -tkc
    >
    >
    > --
    > https://mail.python.org/mailman/listinfo/python-list
    Igor Korot, Dec 15, 2013
    #10
  11. Mark Lawrence

    MRAB Guest

    On 15/12/2013 22:46, Igor Korot wrote:
    > Tim,
    >
    > On Sun, Dec 15, 2013 at 4:29 AM, Tim Chase
    > <> wrote:
    >> On 2013-12-15 06:17, Tim Chase wrote:
    >>>>>>>> conn = sqlite3.connect('x.sqlite',
    >>>>>... detect_types=sqlite3.PARSE_DECLTYPES|sqlite3.PARSE_COLNAMES)
    >>>
    >>> Your example code omitted this one crucial line. Do you specify the
    >>> detect_types parameter to connect()?

    >
    > Yes, I did.
    > This is the beginning of the session:
    >
    >>>> import sqlite3
    >>>> conn = sqlite3.connect('c:\Documents and Settings\Igor.FORDANWORK\Desktop\mydb.db', detect_types = sqlite3.PARSE_DECLTYPES|sqlite3.PARSE_COLNAMES)

    >

    When writing paths on Windows, it's a good idea to use raw string
    literals or slashes instead of backslashes:

    conn = sqlite3.connect(r'c:\Documents and
    Settings\Igor.FORDANWORK\Desktop\mydb.db', detect_types =
    sqlite3.PARSE_DECLTYPES|sqlite3.PARSE_COLNAMES)

    or:

    conn = sqlite3.connect('c:/Documents and
    Settings/Igor.FORDANWORK/Desktop/mydb.db', detect_types =
    sqlite3.PARSE_DECLTYPES|sqlite3.PARSE_COLNAMES)

    > Also please note that you session was missing the cursor creation command. ;-)
    >
    > Thank you.
    >
    >>
    >> It's really the PARSE_DECLTYPES that is important.
    >>
    >> http://docs.python.org/2/library/sqlite3.html#sqlite3.PARSE_DECLTYPES
    >>
    MRAB, Dec 16, 2013
    #11
  12. Mark Lawrence

    Igor Korot Guest

    Hi,

    On Sun, Dec 15, 2013 at 4:58 PM, MRAB <> wrote:
    > On 15/12/2013 22:46, Igor Korot wrote:
    >>
    >> Tim,
    >>
    >> On Sun, Dec 15, 2013 at 4:29 AM, Tim Chase
    >> <> wrote:
    >>>
    >>> On 2013-12-15 06:17, Tim Chase wrote:
    >>>>>>>>>
    >>>>>>>>> conn = sqlite3.connect('x.sqlite',
    >>>>>>
    >>>>>> ... detect_types=sqlite3.PARSE_DECLTYPES|sqlite3.PARSE_COLNAMES)
    >>>>
    >>>>
    >>>> Your example code omitted this one crucial line. Do you specify the
    >>>> detect_types parameter to connect()?

    >>
    >>
    >> Yes, I did.
    >> This is the beginning of the session:
    >>
    >>>>> import sqlite3
    >>>>> conn = sqlite3.connect('c:\Documents and
    >>>>> Settings\Igor.FORDANWORK\Desktop\mydb.db', detect_types =
    >>>>> sqlite3.PARSE_DECLTYPES|sqlite3.PARSE_COLNAMES)

    >>
    >>

    > When writing paths on Windows, it's a good idea to use raw string
    > literals or slashes instead of backslashes:
    >
    > conn = sqlite3.connect(r'c:\Documents and
    > Settings\Igor.FORDANWORK\Desktop\mydb.db', detect_types =
    > sqlite3.PARSE_DECLTYPES|sqlite3.PARSE_COLNAMES)
    >
    > or:
    >
    > conn = sqlite3.connect('c:/Documents and
    > Settings/Igor.FORDANWORK/Desktop/mydb.db', detect_types =
    > sqlite3.PARSE_DECLTYPES|sqlite3.PARSE_COLNAMES)


    So, how do I convert my string to one of those?
    I realized I can just do replace '/' to '\', but is there a better alternative?

    Thank you.


    >
    >
    >> Also please note that you session was missing the cursor creation command.
    >> ;-)
    >>
    >> Thank you.
    >>
    >>>
    >>> It's really the PARSE_DECLTYPES that is important.
    >>>
    >>> http://docs.python.org/2/library/sqlite3.html#sqlite3.PARSE_DECLTYPES
    >>>

    >
    > --
    > https://mail.python.org/mailman/listinfo/python-list
    Igor Korot, Dec 16, 2013
    #12
  13. On Mon, Dec 16, 2013 at 1:43 PM, Igor Korot <> wrote:
    > So, how do I convert my string to one of those?
    > I realized I can just do replace '/' to '\', but is there a better alternative?


    The path is exactly the same, whether you use forward slashes or
    backslashes, on Windows. Most of the world uses backslashes for
    Windows paths, but forward slashes do work fine (and that's not a
    Python thing - the underlying APIs work with / just fine too). So you
    don't have to actually replace anything.

    >>> len(open("c:/Festival2013.txt").read())

    79137

    ChrisA
    Chris Angelico, Dec 16, 2013
    #13
  14. Mark Lawrence

    Dave Angel Guest

    On Sun, 15 Dec 2013 18:43:53 -0800, Igor Korot <>
    wrote:
    > On Sun, Dec 15, 2013 at 4:58 PM, MRAB <>

    wrote:




    > > When writing paths on Windows, it's a good idea to use raw string
    > > literals or slashes instead of backslashes:
    > >
    > > conn = sqlite3.connect(r'c:\Documents and
    > > Settings\Igor.FORDANWORK\Desktop\mydb.db', detect_types =
    > > sqlite3.PARSE_DECLTYPES|sqlite3.PARSE_COLNAMES)
    > >
    > > or:
    > >
    > > conn = sqlite3.connect('c:/Documents and
    > > Settings/Igor.FORDANWORK/Desktop/mydb.db', detect_types =
    > > sqlite3.PARSE_DECLTYPES|sqlite3.PARSE_COLNAMES)


    > So, how do I convert my string to one of those?
    > I realized I can just do replace '/' to '\', but is there a better

    alternative?

    The "conversion" is done with a text editor. The string literals you
    had were just wrong. By the time they have been turned into strings
    it's too late to algorithmically recover your intended string values.

    --
    DaveA
    Dave Angel, Dec 16, 2013
    #14
  15. Mark Lawrence

    Igor Korot Guest

    Hi,
    I have the same result even with:

    sqlite3.connect(r'...')

    Any other alternatives?

    Thank you.


    On Sun, Dec 15, 2013 at 4:58 PM, MRAB <> wrote:
    > On 15/12/2013 22:46, Igor Korot wrote:
    >>
    >> Tim,
    >>
    >> On Sun, Dec 15, 2013 at 4:29 AM, Tim Chase
    >> <> wrote:
    >>>
    >>> On 2013-12-15 06:17, Tim Chase wrote:
    >>>>>>>>>
    >>>>>>>>> conn = sqlite3.connect('x.sqlite',
    >>>>>>
    >>>>>> ... detect_types=sqlite3.PARSE_DECLTYPES|sqlite3.PARSE_COLNAMES)
    >>>>
    >>>>
    >>>> Your example code omitted this one crucial line. Do you specify the
    >>>> detect_types parameter to connect()?

    >>
    >>
    >> Yes, I did.
    >> This is the beginning of the session:
    >>
    >>>>> import sqlite3
    >>>>> conn = sqlite3.connect('c:\Documents and
    >>>>> Settings\Igor.FORDANWORK\Desktop\mydb.db', detect_types =
    >>>>> sqlite3.PARSE_DECLTYPES|sqlite3.PARSE_COLNAMES)

    >>
    >>

    > When writing paths on Windows, it's a good idea to use raw string
    > literals or slashes instead of backslashes:
    >
    > conn = sqlite3.connect(r'c:\Documents and
    > Settings\Igor.FORDANWORK\Desktop\mydb.db', detect_types =
    > sqlite3.PARSE_DECLTYPES|sqlite3.PARSE_COLNAMES)
    >
    > or:
    >
    > conn = sqlite3.connect('c:/Documents and
    > Settings/Igor.FORDANWORK/Desktop/mydb.db', detect_types =
    > sqlite3.PARSE_DECLTYPES|sqlite3.PARSE_COLNAMES)
    >
    >
    >> Also please note that you session was missing the cursor creation command.
    >> ;-)
    >>
    >> Thank you.
    >>
    >>>
    >>> It's really the PARSE_DECLTYPES that is important.
    >>>
    >>> http://docs.python.org/2/library/sqlite3.html#sqlite3.PARSE_DECLTYPES
    >>>

    >
    > --
    > https://mail.python.org/mailman/listinfo/python-list
    Igor Korot, Dec 16, 2013
    #15
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Igor Korot

    Eliminate "extra" variable

    Igor Korot, Dec 6, 2013, in forum: Python
    Replies:
    0
    Views:
    60
    Igor Korot
    Dec 6, 2013
  2. Peter Otten

    Re: Eliminate "extra" variable

    Peter Otten, Dec 8, 2013, in forum: Python
    Replies:
    0
    Views:
    83
    Peter Otten
    Dec 8, 2013
  3. Tim Chase

    Re: Eliminate "extra" variable

    Tim Chase, Dec 8, 2013, in forum: Python
    Replies:
    0
    Views:
    70
    Tim Chase
    Dec 8, 2013
  4. Tim Chase

    Re: Eliminate "extra" variable

    Tim Chase, Dec 8, 2013, in forum: Python
    Replies:
    0
    Views:
    75
    Tim Chase
    Dec 8, 2013
  5. Tim Chase

    Re: Eliminate "extra" variable

    Tim Chase, Dec 8, 2013, in forum: Python
    Replies:
    0
    Views:
    62
    Tim Chase
    Dec 8, 2013
Loading...

Share This Page