semi-Newbie question

[len]

Hi all

I have a file that I receive from another party which is basicly a csv
file containing the following type of information;

Tagname Scope Value
"first_name","POL01","John"
"last_name","POL01","Doe"
"birthday","POL01","04/03/61"
etc

I need to convert this file info into my file format used in my
application.

I have been given a file from the other company that gives me all of
the tagname that could be used and their scope. I want to build a
table which would have all of the various tagnames, scope, and put a
fieldname equivalent in the fieldname in my file structure in my
application. Then read through the vendors csv file index into my
table file and get the field name of where to move the data into my
data structure.

Here is my question? basicly I need to get the data referenced by
fieldname variable in my table and then use that data as a variable
name in a python assignment statement. Thus changing the actual python
code at each iteration through the lines in the csv file.

for i in tagfile
find tagname in tagtable

*** the following line of code would change through each iteration ***
myfirst = value

*** next iteration
mylast = value

*** next iteration
mybirth = value

etc

I hope this make sense. I remember seeing something like this
somewhere.

Any help appreciated.

Len Sumnler
Unique Insurance

 

[davideugenewarren]

Do you absolutely need to use variables? A dictionary would serve if
each case has a unique identifier.

client_info_dict = {}

for i in tagfile:
tagname,scope,value = i.replace('"','').split(',') # split fields,
strip redundant characters
client_info_dict.setdefault(scope,{}) # set up an empty nested
dict for the unique ID
client_info_dict[scope][tagname] = value # set the tagname's value

Then client info can be retrieved from the dictionary using unique IDs
and stereotyped tags (with .get() if some tags are not always present).
I won't make any claims about the efficiency of this approach, but it
works for me.

 

[danielx]

Hi all

I have a file that I receive from another party which is basicly a csv
file containing the following type of information;

Python has a csv module. I'm not sure if you're already using that or
if it would be useful to you:

http://docs.python.org/lib/module-csv.html

Tagname Scope Value
"first_name","POL01","John"
"last_name","POL01","Doe"
"birthday","POL01","04/03/61"
etc

I need to convert this file info into my file format used in my
application.

I have been given a file from the other company that gives me all of
the tagname that could be used and their scope. I want to build a

So, a tag name is like a field name, right? What's a scope?

table which would have all of the various tagnames, scope, and put a
fieldname equivalent in the fieldname in my file structure in my
application. Then read through the vendors csv file index into my
table file and get the field name of where to move the data into my
data structure.

Here is my question? basicly I need to get the data referenced by
fieldname variable in my table and then use that data as a variable
name in a python assignment statement. Thus changing the actual python
code at each iteration through the lines in the csv file.

I'm not sure that you need to have a different statement execute during
each loop. Why not have something like this:

for line in tagfile:
foreignTag, scope, value = parse(line)
currentRow[ myEqvTag(foreignTag) ] = value
## do you want to do something with the scope information?

If you really wanted to, you could use exec/eval and construct a string
you want to execute/evaluate, but people tend to look down on that. At
least one of the reasons is that it would be slower. Just for
completeness, maybe this is what you want:

for line in tagfile:
ft, sc, v = parse(line)
exec "%s = %s" % ( myEqvTag( ft ), v )

Is this something like what you wanted? I'm not quite sure what you're
asking... More code might help.

Good Luck!

 

[len]

I appoligize I don't think I have done a very good job of explaining my
problem.

I work in the an insurance company that sells non standard auto. We
have an in house policy management system that uses MySQL for the data
storage.

An individual policy consists of a policy header and 1 to n drivers, 1
to n vehicles and 1 to n coverage records per vehicle.

A third party company will sell a policy and place it with our company.
They collect all of the information on their system and then send this
information to us as a CSV file (one file per policy). **This CSV file
is like an XML file and is formated as followes;
1st field is a tagname
2nd field is a scope
3rd field is a value

see example

"totalpolicypremium","pol0","1584"
"quotenumber","pol0","5"
"address1","pol0","123 Testing Street"
"address2","pol0",""
"apartmentnumber","pol0",""
"cellphone","pol0","( ) -"
....
"annualmiles","car1","0"
"antilock","car1","A"
"antitheft","car1","1"
"bodytype","car1","4D"
"buybackpip","car1","N"
"carphone","car1","N"
"city","car1","ALEXANDRIA"
"coaccdeathlimit","car1","0"
....
"annualmiles","car2","0"
"antilock","car2","N"
"antitheft","car2","1"
"bodytype","car2","4D"
"buybackpip","car2","N"
"carphone","car2","N"
"city","car2","ALEXANDRIA"
....
"address1","drv1","123 Testing Street"
"address2","drv1",""
"agerated","drv1","0"
"banklienjudgstat","drv1","N"
"cellphone","drv1","( ) -"
"city","drv1","Testing City"
"cluestatus","drv1","N"

etc

The third party company sent me a file that contained all of their
valid tagnames and scope which I then took and create a crossreference
file with three fields:

xreffile
tagname (key)
scopy
SQL_fieldname

The program I am writing is nothing more than a conversion program to
take the value out of the CSV file and map it into the appropriate
field in my SQL files. Rather than creating some huge if than else
(there are over 1000 tagnames) I created the xreffile.

Now when I read a record from the tagfile I use the data in the tagname
field to lookup the tagname in my xreffile. The data in the
SQL_fieldname is the fieldname in my SQL files I want to place the data
from the tagfile in the tagfile.value field into this field in my SQL
files;

data referenced by(xreffile.SQL_fieldname) = tagfile.value

what I see as the problem is I want to use what is the data reference
by xreffile.SQL.fieldname and now make it part of the python code as a
reference variable in an assignement code statement.

I hope this helps

Len Sumnler
Unique Insurance

Do you absolutely need to use variables? A dictionary would serve if
each case has a unique identifier.

client_info_dict = {}

for i in tagfile:
tagname,scope,value = i.replace('"','').split(',') # split fields,
strip redundant characters
client_info_dict.setdefault(scope,{}) # set up an empty nested
dict for the unique ID
client_info_dict[scope][tagname] = value # set the tagname's value

Then client info can be retrieved from the dictionary using unique IDs
and stereotyped tags (with .get() if some tags are not always present).
I won't make any claims about the efficiency of this approach, but it
works for me.

Hi all

I have a file that I receive from another party which is basicly a csv
file containing the following type of information;

Tagname Scope Value
"first_name","POL01","John"
"last_name","POL01","Doe"
"birthday","POL01","04/03/61"
etc

I need to convert this file info into my file format used in my
application.

I have been given a file from the other company that gives me all of
the tagname that could be used and their scope. I want to build a
table which would have all of the various tagnames, scope, and put a
fieldname equivalent in the fieldname in my file structure in my
application. Then read through the vendors csv file index into my
table file and get the field name of where to move the data into my
data structure.

Here is my question? basicly I need to get the data referenced by
fieldname variable in my table and then use that data as a variable
name in a python assignment statement. Thus changing the actual python
code at each iteration through the lines in the csv file.

for i in tagfile
find tagname in tagtable

*** the following line of code would change through each iteration ***
myfirst = value

*** next iteration
mylast = value

*** next iteration
mybirth = value

etc

I hope this make sense. I remember seeing something like this
somewhere.

Any help appreciated.

Len Sumnler
Unique Insurance

 

[John Machin]

Hi all

I have a file that I receive from another party which is basicly a csv
file containing the following type of information;

Tagname Scope Value
"first_name","POL01","John"
"last_name","POL01","Doe"
"birthday","POL01","04/03/61"
etc

I need to convert this file info into my file format used in my
application.

I have been given a file from the other company that gives me all of
the tagname that could be used and their scope. I want to build a
table which would have all of the various tagnames, scope, and put a
fieldname equivalent in the fieldname in my file structure in my
application. Then read through the vendors csv file index into my
table file and get the field name of where to move the data into my
data structure.

Here is my question? basicly I need to get the data referenced by
fieldname variable in my table and then use that data as a variable
name in a python assignment statement. Thus changing the actual python
code at each iteration through the lines in the csv file.

for i in tagfile
find tagname in tagtable

*** the following line of code would change through each iteration ***
myfirst = value

*** next iteration
mylast = value

*** next iteration
mybirth = value

etc

I hope this make sense. I remember seeing something like this
somewhere.

You need to define the problem much better than you have in your two
postings so far. Give a small realistic sample of input file and the
relevant parts of the xref file plus (the main missing component so
far) the actual OUTPUT that you desire from that input.

Then we can talk about implementatiion. However while we're waiting,
wtite this on your whiteboard:

(1) *DO* use the csv module; DIY approaches blow up spectacularly when
presented with data which has embedded commas and quotes, and don't say
it can't happen because it does.

(2) *DON'T* pay any attention to suggestions that you should use exec
or eval. The likelihood that your problem *needs* that "the following
line of code would change through each iteration" is very small. The
likelihood that the solution can be written in a straight-forward
manner is high.

What are your "SQL files"? Do you mean insert scripts that you will use
to inject the transformed incoming into a database, or something else?
Again, a short *example* of what you are referring to would help.

HTH,
John

 

[Cameron Laird]

I appoligize I don't think I have done a very good job of explaining my
problem. .
.
.
The program I am writing is nothing more than a conversion program to
take the value out of the CSV file and map it into the appropriate
field in my SQL files. Rather than creating some huge if than else
(there are over 1000 tagnames) I created the xreffile.

Now when I read a record from the tagfile I use the data in the tagname
field to lookup the tagname in my xreffile. The data in the
SQL_fieldname is the fieldname in my SQL files I want to place the data
from the tagfile in the tagfile.value field into this field in my SQL
files;

data referenced by(xreffile.SQL_fieldname) = tagfile.value

what I see as the problem is I want to use what is the data reference
by xreffile.SQL.fieldname and now make it part of the python code as a
reference variable in an assignement code statement.

.
.
.
1. Take Daniel Wong's advice, elsewhere in this thread,
and use the Python CSV module.
2. "what I see as the problem is I want ...": what you
want *is* rather a problem, because it's a troublesome
way to achieve what I understand to be your larger
aims. It was certainly good that you didn't create
"some huge if than else".

Once you have an SQL_fieldname, and a tagfile.value,
what do you want to do? Are you stuffing data into
an SQL table? Continuing on with Python computations?
In almost any case, it sounds as though you will profit
greatly from study of Python's dictionaries <URL:
http://www.developer.com/lang/other/article.php/630721 >
<URL: http://www.diveintopython.org/getting_to_know_python/dictionaries.html >.

 

[len]

Thank all for your reply.

I will try again to state the problem.

I have three files.

1. Tagfile - This file contains data in a CSV format each record in
the file contains three fields 'Tagname', 'Scope', and 'Value' and
exampe of this data file follows;
"totalpolicypremium","pol0","1584"
"quotenumber","pol0","5"
"address1","pol0","123 Testing Street"
"address2","pol0",""
"apartmentnumber","pol0",""
"cellphone","pol0","( ) -"
....
"annualmiles","car1","0"
"antilock","car1","A"
"antitheft","car1","1"
"bodytype","car1","4D"
"buybackpip","car1","N"
"carphone","car1","N"
"city","car1","ALEXANDRIA"
"coaccdeathlimit","car1","0"
....
"annualmiles","car2","0"
"antilock","car2","N"
"antitheft","car2","1"
"bodytype","car2","4D"
"buybackpip","car2","N"
"carphone","car2","N"
"city","car2","ALEXANDRIA"
....
"address1","drv1","123 Testing Street"
"address2","drv1",""
"agerated","drv1","0"
"banklienjudgstat","drv1","N"
"cellphone","drv1","( ) -"
"city","drv1","Testing City"
"cluestatus","drv1","N"

etc

I have already written the code that can pass through this file using
the CSV module and extract the data as I need it from this file, no
problem here.

2. TagToSQL - This is a file which I created which also contains 3
fields as follows
'theirTagname', 'theirScope', and 'mySQLfieldname' and acts as a
crossreference file between the Tagfile and my SQL file example of
TagToSQL this file has a primary index on theirTagname;

"address1","POL1","bnd.addr1"
"address2","POL1","bnd.addr2"
"appartmentnumber","POL1","bnd.apptno"

etc
3. Binder - This is the primary Policy header file and is in MySQL
this file contains information such as;
bnd.policyno
bnd.last
bnd.first
bnd.addr1
bnd.addr2
bnd.city
bnd.state

etc

Now most of my coding experience is in compiled languages such as
cobol, c, assembler etc

I have all of the file access code completed as far as reading through
the CSV file and indexing into my TagToSQL file and writing to my SQL
files the only problem I have is how to create what I believe to be the
couple of lines of code which would allow me to move the date in
'Value' from the Tagfile into my SQL file using the data in the
TagToSQL field mySQLfieldname.

I have done some more reading and I think the code I need is as
follows;

mycode = "TagToSQL['mySQLfieldname'] = Tagfile['Value']"
exec mycode

This is very new to me because I don't believe this can be done in a
compiled language or at least not as easily as in an interpeted
language like Python.

I hope this clarifies the problem

Len Sumnler
Unique Insurance

 

[Marc 'BlackJack' Rintsch]

mycode = "TagToSQL['mySQLfieldname'] = Tagfile['Value']"
exec mycode

Why do you use ``exec`` here? Why not simply::

TagToSQL['mySQLfieldname'] = Tagfile['Value']

Ciao,
Marc 'BlackJack' Rintsch

 

[Cameron Laird]

.
.
.

I have done some more reading and I think the code I need is as
follows;

mycode = "TagToSQL['mySQLfieldname'] = Tagfile['Value']"
exec mycode

This is very new to me because I don't believe this can be done in a
compiled language or at least not as easily as in an interpeted
language like Python.

I hope this clarifies the problem

.
.
.
I don't understand how

TagToSQL[mySQLfieldname] = Tagfile[Value]

fails to meet requirements that

mycode = "TagToSQL['mySQLfieldname'] = Tagfile['Value']"
exec mycode

satisfies.

This thread confuses me. Maybe you already have all the answers
you seek. If not, I recommend that you simplify--perhaps work
through a single example datum in detail. In the meantime, I
applaud your judgment that you can achieve what you're after with
table lookups and such rather than the thousand-way if-else at
which you hinted at least once.

 

[len]

Sample and test code shows you are correct.

tpsFile - is really the SQL file I will be inserting new policy records
into

tagFile - is a CVS file containing all of the information for a new
policy in an XMLish fashion (one record per filed of the policy) I will
receive from a third party

tagIdxFile - is just a file that where the data from the tagFile should
be mapped into the tpsFile

CODE

tpsFile = {'tpsFirstName' : 'Kate', 'tpsLastName' : 'Sumner',
'tpsPhone': '532-1234'}
tagFile = {'tagname' : 'tagFirst', 'tagScope' : 'POL0', 'tagValue' :
'Rose'}
tagIdxFile = {'idxtagname' : 'tagFirst', 'idxtpsname' : 'tpsFirstName'}
print tpsFile['tpsFirstName']
tpsFile[tagIdxFile['idxtpsname']] = tagFile['tagValue']
print tpsFile['tpsFirstName']

RESULTS

Kate
Rose

Just a small note: As trivial as this may seem this task was not
possible in the compiled language I work in due to the fact that there
was no way for me to get the data referenced by
tagIdxFile['idxtpsname'] and then use it as a field label on the left
side of the assignment statement because in the compiled language the
left side must be a label and NOT and expression.

Just strengthens my commitment to learn Python.


I would like to thank every one for their help, advice and patients.

Len Sumnler

.
.
.

I have done some more reading and I think the code I need is as
follows;

mycode = "TagToSQL['mySQLfieldname'] = Tagfile['Value']"
exec mycode

This is very new to me because I don't believe this can be done in a
compiled language or at least not as easily as in an interpeted
language like Python.

I hope this clarifies the problem

.
.
.
I don't understand how

TagToSQL[mySQLfieldname] = Tagfile[Value]

fails to meet requirements that

mycode = "TagToSQL['mySQLfieldname'] = Tagfile['Value']"
exec mycode

satisfies.

This thread confuses me. Maybe you already have all the answers
you seek. If not, I recommend that you simplify--perhaps work
through a single example datum in detail. In the meantime, I
applaud your judgment that you can achieve what you're after with
table lookups and such rather than the thousand-way if-else at
which you hinted at least once.

 

[Dennis Lee Bieber]

Just a small note: As trivial as this may seem this task was not
possible in the compiled language I work in due to the fact that there
was no way for me to get the data referenced by
tagIdxFile['idxtpsname'] and then use it as a field label on the left
side of the assignment statement because in the compiled language the
left side must be a label and NOT and expression.

It could probably even be done in FORTRAN... One just needs to know
how to write hash/binary tables and look-up functions to translate a
string data value into an index to a fixed array.

Instead of something like
tagIdxFile["idxtpsname"]

tagIdxFile(idxbyname("idxtpsname"))

or variations.

Is it as easy as Python? No...

One of my college assignments, in the data structures course, was to
implement a hashed-head, multiple-linked, list (I did it in a 70s
BASIC). I've only seen details of such a structure in use once in 30
years -- the disk organization of the Amiga OS.
--
Wulfraed Dennis Lee Bieber KD6MOG
(e-mail address removed) (e-mail address removed)
HTTP://wlfraed.home.netcom.com/
(Bestiaria Support Staff: (e-mail address removed))
HTTP://www.bestiaria.com/