semi-Newbie question

Discussion in 'Python' started by len, Aug 10, 2006.

  1. len

    len Guest

    Hi all

    I have a file that I receive from another party which is basicly a csv
    file containing the following type of information;

    Tagname Scope Value
    "first_name","POL01","John"
    "last_name","POL01","Doe"
    "birthday","POL01","04/03/61"
    etc

    I need to convert this file info into my file format used in my
    application.

    I have been given a file from the other company that gives me all of
    the tagname that could be used and their scope. I want to build a
    table which would have all of the various tagnames, scope, and put a
    fieldname equivalent in the fieldname in my file structure in my
    application. Then read through the vendors csv file index into my
    table file and get the field name of where to move the data into my
    data structure.

    Here is my question? basicly I need to get the data referenced by
    fieldname variable in my table and then use that data as a variable
    name in a python assignment statement. Thus changing the actual python
    code at each iteration through the lines in the csv file.

    for i in tagfile
    find tagname in tagtable

    *** the following line of code would change through each iteration ***
    myfirst = value

    *** next iteration
    mylast = value

    *** next iteration
    mybirth = value

    etc

    I hope this make sense. I remember seeing something like this
    somewhere.

    Any help appreciated.

    Len Sumnler
    Unique Insurance
     
    len, Aug 10, 2006
    #1
    1. Advertising

  2. len

    Guest

    Do you absolutely need to use variables? A dictionary would serve if
    each case has a unique identifier.

    client_info_dict = {}

    for i in tagfile:
    tagname,scope,value = i.replace('"','').split(',') # split fields,
    strip redundant characters
    client_info_dict.setdefault(scope,{}) # set up an empty nested
    dict for the unique ID
    client_info_dict[scope][tagname] = value # set the tagname's value

    Then client info can be retrieved from the dictionary using unique IDs
    and stereotyped tags (with .get() if some tags are not always present).
    I won't make any claims about the efficiency of this approach, but it
    works for me.

    len wrote:
    > Hi all
    >
    > I have a file that I receive from another party which is basicly a csv
    > file containing the following type of information;
    >
    > Tagname Scope Value
    > "first_name","POL01","John"
    > "last_name","POL01","Doe"
    > "birthday","POL01","04/03/61"
    > etc
    >
    > I need to convert this file info into my file format used in my
    > application.
    >
    > I have been given a file from the other company that gives me all of
    > the tagname that could be used and their scope. I want to build a
    > table which would have all of the various tagnames, scope, and put a
    > fieldname equivalent in the fieldname in my file structure in my
    > application. Then read through the vendors csv file index into my
    > table file and get the field name of where to move the data into my
    > data structure.
    >
    > Here is my question? basicly I need to get the data referenced by
    > fieldname variable in my table and then use that data as a variable
    > name in a python assignment statement. Thus changing the actual python
    > code at each iteration through the lines in the csv file.
    >
    > for i in tagfile
    > find tagname in tagtable
    >
    > *** the following line of code would change through each iteration ***
    > myfirst = value
    >
    > *** next iteration
    > mylast = value
    >
    > *** next iteration
    > mybirth = value
    >
    > etc
    >
    > I hope this make sense. I remember seeing something like this
    > somewhere.
    >
    > Any help appreciated.
    >
    > Len Sumnler
    > Unique Insurance
     
    , Aug 10, 2006
    #2
    1. Advertising

  3. len

    danielx Guest

    len wrote:
    > Hi all
    >
    > I have a file that I receive from another party which is basicly a csv
    > file containing the following type of information;


    Python has a csv module. I'm not sure if you're already using that or
    if it would be useful to you:

    http://docs.python.org/lib/module-csv.html

    >
    > Tagname Scope Value
    > "first_name","POL01","John"
    > "last_name","POL01","Doe"
    > "birthday","POL01","04/03/61"
    > etc
    >
    > I need to convert this file info into my file format used in my
    > application.
    >
    > I have been given a file from the other company that gives me all of
    > the tagname that could be used and their scope. I want to build a


    So, a tag name is like a field name, right? What's a scope?

    > table which would have all of the various tagnames, scope, and put a
    > fieldname equivalent in the fieldname in my file structure in my
    > application. Then read through the vendors csv file index into my
    > table file and get the field name of where to move the data into my
    > data structure.
    >
    > Here is my question? basicly I need to get the data referenced by
    > fieldname variable in my table and then use that data as a variable
    > name in a python assignment statement. Thus changing the actual python
    > code at each iteration through the lines in the csv file.


    I'm not sure that you need to have a different statement execute during
    each loop. Why not have something like this:

    for line in tagfile:
    foreignTag, scope, value = parse(line)
    currentRow[ myEqvTag(foreignTag) ] = value
    ## do you want to do something with the scope information?

    If you really wanted to, you could use exec/eval and construct a string
    you want to execute/evaluate, but people tend to look down on that. At
    least one of the reasons is that it would be slower. Just for
    completeness, maybe this is what you want:

    for line in tagfile:
    ft, sc, v = parse(line)
    exec "%s = %s" % ( myEqvTag( ft ), v )

    Is this something like what you wanted? I'm not quite sure what you're
    asking... More code might help.

    Good Luck!

    >
    > for i in tagfile
    > find tagname in tagtable
    >
    > *** the following line of code would change through each iteration ***
    > myfirst = value
    >
    > *** next iteration
    > mylast = value
    >
    > *** next iteration
    > mybirth = value
    >
    > etc
    >
    > I hope this make sense. I remember seeing something like this
    > somewhere.
    >
    > Any help appreciated.
    >
    > Len Sumnler
    > Unique Insurance
     
    danielx, Aug 10, 2006
    #3
  4. len

    len Guest

    I appoligize I don't think I have done a very good job of explaining my
    problem.

    I work in the an insurance company that sells non standard auto. We
    have an in house policy management system that uses MySQL for the data
    storage.

    An individual policy consists of a policy header and 1 to n drivers, 1
    to n vehicles and 1 to n coverage records per vehicle.

    A third party company will sell a policy and place it with our company.
    They collect all of the information on their system and then send this
    information to us as a CSV file (one file per policy). **This CSV file
    is like an XML file and is formated as followes;
    1st field is a tagname
    2nd field is a scope
    3rd field is a value

    see example

    "totalpolicypremium","pol0","1584"
    "quotenumber","pol0","5"
    "address1","pol0","123 Testing Street"
    "address2","pol0",""
    "apartmentnumber","pol0",""
    "cellphone","pol0","( ) -"
    ....
    "annualmiles","car1","0"
    "antilock","car1","A"
    "antitheft","car1","1"
    "bodytype","car1","4D"
    "buybackpip","car1","N"
    "carphone","car1","N"
    "city","car1","ALEXANDRIA"
    "coaccdeathlimit","car1","0"
    ....
    "annualmiles","car2","0"
    "antilock","car2","N"
    "antitheft","car2","1"
    "bodytype","car2","4D"
    "buybackpip","car2","N"
    "carphone","car2","N"
    "city","car2","ALEXANDRIA"
    ....
    "address1","drv1","123 Testing Street"
    "address2","drv1",""
    "agerated","drv1","0"
    "banklienjudgstat","drv1","N"
    "cellphone","drv1","( ) -"
    "city","drv1","Testing City"
    "cluestatus","drv1","N"

    etc

    The third party company sent me a file that contained all of their
    valid tagnames and scope which I then took and create a crossreference
    file with three fields:

    xreffile
    tagname (key)
    scopy
    SQL_fieldname

    The program I am writing is nothing more than a conversion program to
    take the value out of the CSV file and map it into the appropriate
    field in my SQL files. Rather than creating some huge if than else
    (there are over 1000 tagnames) I created the xreffile.

    Now when I read a record from the tagfile I use the data in the tagname
    field to lookup the tagname in my xreffile. The data in the
    SQL_fieldname is the fieldname in my SQL files I want to place the data
    from the tagfile in the tagfile.value field into this field in my SQL
    files;

    data referenced by(xreffile.SQL_fieldname) = tagfile.value

    what I see as the problem is I want to use what is the data reference
    by xreffile.SQL.fieldname and now make it part of the python code as a
    reference variable in an assignement code statement.

    I hope this helps

    Len Sumnler
    Unique Insurance

    wrote:
    > Do you absolutely need to use variables? A dictionary would serve if
    > each case has a unique identifier.
    >
    > client_info_dict = {}
    >
    > for i in tagfile:
    > tagname,scope,value = i.replace('"','').split(',') # split fields,
    > strip redundant characters
    > client_info_dict.setdefault(scope,{}) # set up an empty nested
    > dict for the unique ID
    > client_info_dict[scope][tagname] = value # set the tagname's value
    >
    > Then client info can be retrieved from the dictionary using unique IDs
    > and stereotyped tags (with .get() if some tags are not always present).
    > I won't make any claims about the efficiency of this approach, but it
    > works for me.
    >
    > len wrote:
    > > Hi all
    > >
    > > I have a file that I receive from another party which is basicly a csv
    > > file containing the following type of information;
    > >
    > > Tagname Scope Value
    > > "first_name","POL01","John"
    > > "last_name","POL01","Doe"
    > > "birthday","POL01","04/03/61"
    > > etc
    > >
    > > I need to convert this file info into my file format used in my
    > > application.
    > >
    > > I have been given a file from the other company that gives me all of
    > > the tagname that could be used and their scope. I want to build a
    > > table which would have all of the various tagnames, scope, and put a
    > > fieldname equivalent in the fieldname in my file structure in my
    > > application. Then read through the vendors csv file index into my
    > > table file and get the field name of where to move the data into my
    > > data structure.
    > >
    > > Here is my question? basicly I need to get the data referenced by
    > > fieldname variable in my table and then use that data as a variable
    > > name in a python assignment statement. Thus changing the actual python
    > > code at each iteration through the lines in the csv file.
    > >
    > > for i in tagfile
    > > find tagname in tagtable
    > >
    > > *** the following line of code would change through each iteration ***
    > > myfirst = value
    > >
    > > *** next iteration
    > > mylast = value
    > >
    > > *** next iteration
    > > mybirth = value
    > >
    > > etc
    > >
    > > I hope this make sense. I remember seeing something like this
    > > somewhere.
    > >
    > > Any help appreciated.
    > >
    > > Len Sumnler
    > > Unique Insurance
     
    len, Aug 10, 2006
    #4
  5. len

    John Machin Guest

    len wrote:
    > Hi all
    >
    > I have a file that I receive from another party which is basicly a csv
    > file containing the following type of information;
    >
    > Tagname Scope Value
    > "first_name","POL01","John"
    > "last_name","POL01","Doe"
    > "birthday","POL01","04/03/61"
    > etc
    >
    > I need to convert this file info into my file format used in my
    > application.
    >
    > I have been given a file from the other company that gives me all of
    > the tagname that could be used and their scope. I want to build a
    > table which would have all of the various tagnames, scope, and put a
    > fieldname equivalent in the fieldname in my file structure in my
    > application. Then read through the vendors csv file index into my
    > table file and get the field name of where to move the data into my
    > data structure.
    >
    > Here is my question? basicly I need to get the data referenced by
    > fieldname variable in my table and then use that data as a variable
    > name in a python assignment statement. Thus changing the actual python
    > code at each iteration through the lines in the csv file.
    >
    > for i in tagfile
    > find tagname in tagtable
    >
    > *** the following line of code would change through each iteration ***
    > myfirst = value
    >
    > *** next iteration
    > mylast = value
    >
    > *** next iteration
    > mybirth = value
    >
    > etc
    >
    > I hope this make sense. I remember seeing something like this
    > somewhere.
    >


    You need to define the problem much better than you have in your two
    postings so far. Give a small realistic sample of input file and the
    relevant parts of the xref file plus (the main missing component so
    far) the actual OUTPUT that you desire from that input.

    Then we can talk about implementatiion. However while we're waiting,
    wtite this on your whiteboard:

    (1) *DO* use the csv module; DIY approaches blow up spectacularly when
    presented with data which has embedded commas and quotes, and don't say
    it can't happen because it does.

    (2) *DON'T* pay any attention to suggestions that you should use exec
    or eval. The likelihood that your problem *needs* that "the following
    line of code would change through each iteration" is very small. The
    likelihood that the solution can be written in a straight-forward
    manner is high.

    What are your "SQL files"? Do you mean insert scripts that you will use
    to inject the transformed incoming into a database, or something else?
    Again, a short *example* of what you are referring to would help.

    HTH,
    John
     
    John Machin, Aug 10, 2006
    #5
  6. In article <>,
    len <> wrote:
    >I appoligize I don't think I have done a very good job of explaining my
    >problem.

    .
    .
    .
    >The program I am writing is nothing more than a conversion program to
    >take the value out of the CSV file and map it into the appropriate
    >field in my SQL files. Rather than creating some huge if than else
    >(there are over 1000 tagnames) I created the xreffile.
    >
    >Now when I read a record from the tagfile I use the data in the tagname
    >field to lookup the tagname in my xreffile. The data in the
    >SQL_fieldname is the fieldname in my SQL files I want to place the data
    >from the tagfile in the tagfile.value field into this field in my SQL
    >files;
    >
    >data referenced by(xreffile.SQL_fieldname) = tagfile.value
    >
    >what I see as the problem is I want to use what is the data reference
    >by xreffile.SQL.fieldname and now make it part of the python code as a
    >reference variable in an assignement code statement.

    .
    .
    .
    1. Take Daniel Wong's advice, elsewhere in this thread,
    and use the Python CSV module.
    2. "what I see as the problem is I want ...": what you
    want *is* rather a problem, because it's a troublesome
    way to achieve what I understand to be your larger
    aims. It was certainly good that you didn't create
    "some huge if than else".

    Once you have an SQL_fieldname, and a tagfile.value,
    what do you want to do? Are you stuffing data into
    an SQL table? Continuing on with Python computations?
    In almost any case, it sounds as though you will profit
    greatly from study of Python's dictionaries <URL:
    http://www.developer.com/lang/other/article.php/630721 >
    <URL: http://www.diveintopython.org/getting_to_know_python/dictionaries.html >.
     
    Cameron Laird, Aug 10, 2006
    #6
  7. len

    len Guest

    Thank all for your reply.

    I will try again to state the problem.

    I have three files.

    1. Tagfile - This file contains data in a CSV format each record in
    the file contains three fields 'Tagname', 'Scope', and 'Value' and
    exampe of this data file follows;
    "totalpolicypremium","pol0","1584"
    "quotenumber","pol0","5"
    "address1","pol0","123 Testing Street"
    "address2","pol0",""
    "apartmentnumber","pol0",""
    "cellphone","pol0","( ) -"
    ....
    "annualmiles","car1","0"
    "antilock","car1","A"
    "antitheft","car1","1"
    "bodytype","car1","4D"
    "buybackpip","car1","N"
    "carphone","car1","N"
    "city","car1","ALEXANDRIA"
    "coaccdeathlimit","car1","0"
    ....
    "annualmiles","car2","0"
    "antilock","car2","N"
    "antitheft","car2","1"
    "bodytype","car2","4D"
    "buybackpip","car2","N"
    "carphone","car2","N"
    "city","car2","ALEXANDRIA"
    ....
    "address1","drv1","123 Testing Street"
    "address2","drv1",""
    "agerated","drv1","0"
    "banklienjudgstat","drv1","N"
    "cellphone","drv1","( ) -"
    "city","drv1","Testing City"
    "cluestatus","drv1","N"

    etc

    I have already written the code that can pass through this file using
    the CSV module and extract the data as I need it from this file, no
    problem here.

    2. TagToSQL - This is a file which I created which also contains 3
    fields as follows
    'theirTagname', 'theirScope', and 'mySQLfieldname' and acts as a
    crossreference file between the Tagfile and my SQL file example of
    TagToSQL this file has a primary index on theirTagname;

    "address1","POL1","bnd.addr1"
    "address2","POL1","bnd.addr2"
    "appartmentnumber","POL1","bnd.apptno"

    etc
    3. Binder - This is the primary Policy header file and is in MySQL
    this file contains information such as;
    bnd.policyno
    bnd.last
    bnd.first
    bnd.addr1
    bnd.addr2
    bnd.city
    bnd.state

    etc

    Now most of my coding experience is in compiled languages such as
    cobol, c, assembler etc

    I have all of the file access code completed as far as reading through
    the CSV file and indexing into my TagToSQL file and writing to my SQL
    files the only problem I have is how to create what I believe to be the
    couple of lines of code which would allow me to move the date in
    'Value' from the Tagfile into my SQL file using the data in the
    TagToSQL field mySQLfieldname.

    I have done some more reading and I think the code I need is as
    follows;

    mycode = "TagToSQL['mySQLfieldname'] = Tagfile['Value']"
    exec mycode

    This is very new to me because I don't believe this can be done in a
    compiled language or at least not as easily as in an interpeted
    language like Python.

    I hope this clarifies the problem

    Len Sumnler
    Unique Insurance

    Cameron Laird wrote:
    > In article <>,
    > len <> wrote:
    > >I appoligize I don't think I have done a very good job of explaining my
    > >problem.

    > .
    > .
    > .
    > >The program I am writing is nothing more than a conversion program to
    > >take the value out of the CSV file and map it into the appropriate
    > >field in my SQL files. Rather than creating some huge if than else
    > >(there are over 1000 tagnames) I created the xreffile.
    > >
    > >Now when I read a record from the tagfile I use the data in the tagname
    > >field to lookup the tagname in my xreffile. The data in the
    > >SQL_fieldname is the fieldname in my SQL files I want to place the data
    > >from the tagfile in the tagfile.value field into this field in my SQL
    > >files;
    > >
    > >data referenced by(xreffile.SQL_fieldname) = tagfile.value
    > >
    > >what I see as the problem is I want to use what is the data reference
    > >by xreffile.SQL.fieldname and now make it part of the python code as a
    > >reference variable in an assignement code statement.

    > .
    > .
    > .
    > 1. Take Daniel Wong's advice, elsewhere in this thread,
    > and use the Python CSV module.
    > 2. "what I see as the problem is I want ...": what you
    > want *is* rather a problem, because it's a troublesome
    > way to achieve what I understand to be your larger
    > aims. It was certainly good that you didn't create
    > "some huge if than else".
    >
    > Once you have an SQL_fieldname, and a tagfile.value,
    > what do you want to do? Are you stuffing data into
    > an SQL table? Continuing on with Python computations?
    > In almost any case, it sounds as though you will profit
    > greatly from study of Python's dictionaries <URL:
    > http://www.developer.com/lang/other/article.php/630721 >
    > <URL: http://www.diveintopython.org/getting_to_know_python/dictionaries.html >.
     
    len, Aug 11, 2006
    #7
  8. In <>, len wrote:

    > mycode = "TagToSQL['mySQLfieldname'] = Tagfile['Value']"
    > exec mycode


    Why do you use ``exec`` here? Why not simply::

    TagToSQL['mySQLfieldname'] = Tagfile['Value']

    Ciao,
    Marc 'BlackJack' Rintsch
     
    Marc 'BlackJack' Rintsch, Aug 11, 2006
    #8
  9. In article <>,
    len <> wrote:
    .
    .
    .
    >I have done some more reading and I think the code I need is as
    >follows;
    >
    >mycode = "TagToSQL['mySQLfieldname'] = Tagfile['Value']"
    >exec mycode
    >
    >This is very new to me because I don't believe this can be done in a
    >compiled language or at least not as easily as in an interpeted
    >language like Python.
    >
    >I hope this clarifies the problem

    .
    .
    .
    I don't understand how

    TagToSQL[mySQLfieldname] = Tagfile[Value]

    fails to meet requirements that

    mycode = "TagToSQL['mySQLfieldname'] = Tagfile['Value']"
    exec mycode

    satisfies.

    This thread confuses me. Maybe you already have all the answers
    you seek. If not, I recommend that you simplify--perhaps work
    through a single example datum in detail. In the meantime, I
    applaud your judgment that you can achieve what you're after with
    table lookups and such rather than the thousand-way if-else at
    which you hinted at least once.
     
    Cameron Laird, Aug 12, 2006
    #9
  10. len

    len Guest

    Sample and test code shows you are correct.

    tpsFile - is really the SQL file I will be inserting new policy records
    into

    tagFile - is a CVS file containing all of the information for a new
    policy in an XMLish fashion (one record per filed of the policy) I will
    receive from a third party

    tagIdxFile - is just a file that where the data from the tagFile should
    be mapped into the tpsFile

    CODE

    tpsFile = {'tpsFirstName' : 'Kate', 'tpsLastName' : 'Sumner',
    'tpsPhone': '532-1234'}
    tagFile = {'tagname' : 'tagFirst', 'tagScope' : 'POL0', 'tagValue' :
    'Rose'}
    tagIdxFile = {'idxtagname' : 'tagFirst', 'idxtpsname' : 'tpsFirstName'}
    print tpsFile['tpsFirstName']
    tpsFile[tagIdxFile['idxtpsname']] = tagFile['tagValue']
    print tpsFile['tpsFirstName']

    RESULTS

    >>>

    Kate
    Rose
    >>>


    Just a small note: As trivial as this may seem this task was not
    possible in the compiled language I work in due to the fact that there
    was no way for me to get the data referenced by
    tagIdxFile['idxtpsname'] and then use it as a field label on the left
    side of the assignment statement because in the compiled language the
    left side must be a label and NOT and expression.

    Just strengthens my commitment to learn Python.


    I would like to thank every one for their help, advice and patients.

    Len Sumnler
    Cameron Laird wrote:
    > In article <>,
    > len <> wrote:
    > .
    > .
    > .
    > >I have done some more reading and I think the code I need is as
    > >follows;
    > >
    > >mycode = "TagToSQL['mySQLfieldname'] = Tagfile['Value']"
    > >exec mycode
    > >
    > >This is very new to me because I don't believe this can be done in a
    > >compiled language or at least not as easily as in an interpeted
    > >language like Python.
    > >
    > >I hope this clarifies the problem

    > .
    > .
    > .
    > I don't understand how
    >
    > TagToSQL[mySQLfieldname] = Tagfile[Value]
    >
    > fails to meet requirements that
    >
    > mycode = "TagToSQL['mySQLfieldname'] = Tagfile['Value']"
    > exec mycode
    >
    > satisfies.
    >
    > This thread confuses me. Maybe you already have all the answers
    > you seek. If not, I recommend that you simplify--perhaps work
    > through a single example datum in detail. In the meantime, I
    > applaud your judgment that you can achieve what you're after with
    > table lookups and such rather than the thousand-way if-else at
    > which you hinted at least once.
     
    len, Aug 13, 2006
    #10
  11. On 13 Aug 2006 09:16:49 -0700, "len" <> declaimed the
    following in comp.lang.python:

    > Just a small note: As trivial as this may seem this task was not
    > possible in the compiled language I work in due to the fact that there
    > was no way for me to get the data referenced by
    > tagIdxFile['idxtpsname'] and then use it as a field label on the left
    > side of the assignment statement because in the compiled language the
    > left side must be a label and NOT and expression.
    >

    It could probably even be done in FORTRAN... One just needs to know
    how to write hash/binary tables and look-up functions to translate a
    string data value into an index to a fixed array.

    Instead of something like
    tagIdxFile["idxtpsname"]

    tagIdxFile(idxbyname("idxtpsname"))

    or variations.

    Is it as easy as Python? No...

    One of my college assignments, in the data structures course, was to
    implement a hashed-head, multiple-linked, list (I did it in a 70s
    BASIC). I've only seen details of such a structure in use once in 30
    years -- the disk organization of the Amiga OS.
    --
    Wulfraed Dennis Lee Bieber KD6MOG

    HTTP://wlfraed.home.netcom.com/
    (Bestiaria Support Staff: )
    HTTP://www.bestiaria.com/
     
    Dennis Lee Bieber, Aug 13, 2006
    #11
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. =?Utf-8?B?RFQ=?=
    Replies:
    1
    Views:
    1,801
    Brock Allen
    Apr 4, 2005
  2. Alf P. Steinbach
    Replies:
    3
    Views:
    709
    Peter Forthmann
    Jan 30, 2004
  3. mike kreiner
    Replies:
    11
    Views:
    575
    Bengt Richter
    Dec 30, 2004
  4. Replies:
    12
    Views:
    735
  5. James Stroud

    py2app semi-standalone semi-works

    James Stroud, Oct 4, 2006, in forum: Python
    Replies:
    2
    Views:
    735
    James Stroud
    Oct 4, 2006
Loading...

Share This Page