split string at commas respecting quotes when string not in csv format

R

R. David Murray

OK, I've got a little problem that I'd like to ask the assembled minds
for help with. I can write code to parse this, but I'm thinking it may
be possible to do it with regexes. My regex foo isn't that good, so if
anyone is willing to help (or offer an alternate parsing suggestion)
I would be greatful. (This has to be stdlib only, by the way, I
can't introduce any new modules into the application so pyparsing is
not an option.)

The challenge is to turn a string like this:

a=1,b="0234,)#($)@", k="7"

into this:

[("a", "1"), ("b", "0234,)#($)#"), ("k", "7")]
 
J

John Machin

OK, I've got a little problem that I'd like to ask the assembled minds
for help with.  I can write code to parse this, but I'm thinking it may
be possible to do it with regexes.  My regex foo isn't that good, so if
anyone is willing to help (or offer an alternate parsing suggestion)
I would be greatful.  (This has to be stdlib only, by the way, I
can't introduce any new modules into the application so pyparsing is
not an option.)

The challenge is to turn a string like this:

    a=1,b="0234,)#($)@", k="7"

into this:

    [("a", "1"), ("b", "0234,)#($)#"), ("k", "7")]

The challenge is for you to explain unambiguously what you want.

1. a=1 => "1" and k="7" => "7" ... is this a mistake or are the quotes
optional in the original string when not required to protect a comma?

2. What is the rule that explains the transmogrification of @ to # in
your example?

3. Is the input guaranteed to be syntactically correct?

The following should do close enough to what you want; adjust as
appropriate.
>>> import re
>>> s = """a=1,b="0234,)#($)@", k="7" """
>>> rx = re.compile(r'[ ]*(\w+)=([^",]+|"[^"]*")[ ]*(?:,|$)')
>>> rx.findall(s) [('a', '1'), ('b', '"0234,)#($)@"'), ('k', '"7"')]
>>> rx.findall('a=1, *DODGY*SYNTAX* b=2') [('a', '1'), ('b', '2')]
>>>

HTH,
John
 
P

Paul McGuire

OK, I've got a little problem that I'd like to ask the assembled minds
for help with.  I can write code to parse this, but I'm thinking it may
be possible to do it with regexes.  My regex foo isn't that good, so if
anyone is willing to help (or offer an alternate parsing suggestion)
I would be greatful.  (This has to be stdlib only, by the way, I
can't introduce any new modules into the application so pyparsing is
not an option.)

The challenge is to turn a string like this:

    a=1,b="0234,)#($)@", k="7"

into this:

    [("a", "1"), ("b", "0234,)#($)#"), ("k", "7")]

If you must cram all your code into a single source file, then
pyparsing would be problematic. But pyparsing's installation
footprint is really quite small, just a single Python source file. So
if your program spans more than one file, just add pyparsing.py into
the local directory along with everything else.

Then you could write this little parser and be done (note the
differentiation between 1 and "7"):

test = 'a=1,b="0234,)#($)@", k="7"'

from pyparsing import Suppress, Word, alphas, alphanums, \
nums, quotedString, removeQuotes, Group, delimitedList

EQ = Suppress('=')
varname = Word(alphas,alphanums)
integer = Word(nums).setParseAction(lambda t:int(t[0]))
varvalue = integer | quotedString.setParseAction(removeQuotes)
var_assignment = varname("name") + EQ + varvalue("rhs")
expr = delimitedList(Group(var_assignment))

results = expr.parseString(test)
print results.asList()
for assignment in results:
print assignment.name, '<-', repr(assignment.rhs)

Prints:

[['a', 1], ['b', '0234,)#($)@'], ['k', '7']]
a <- 1
b <- '0234,)#($)@'
k <- '7'

-- Paul
 
R

R. David Murray

John Machin said:
OK, I've got a little problem that I'd like to ask the assembled minds
for help with.  I can write code to parse this, but I'm thinking it may
be possible to do it with regexes.  My regex foo isn't that good, so if
anyone is willing to help (or offer an alternate parsing suggestion)
I would be greatful.  (This has to be stdlib only, by the way, I
can't introduce any new modules into the application so pyparsing is
not an option.)

The challenge is to turn a string like this:

    a=1,b="0234,)#($)@", k="7"

into this:

    [("a", "1"), ("b", "0234,)#($)#"), ("k", "7")]

The challenge is for you to explain unambiguously what you want.

1. a=1 => "1" and k="7" => "7" ... is this a mistake or are the quotes
optional in the original string when not required to protect a comma?
optional.

2. What is the rule that explains the transmogrification of @ to # in
your example?

Now that's a mistake :)
3. Is the input guaranteed to be syntactically correct?

If it's not, it's the customer that gets to deal with the error.
The following should do close enough to what you want; adjust as
appropriate.
import re
s = """a=1,b="0234,)#($)@", k="7" """
rx = re.compile(r'[ ]*(\w+)=([^",]+|"[^"]*")[ ]*(?:,|$)')
rx.findall(s) [('a', '1'), ('b', '"0234,)#($)@"'), ('k', '"7"')]
rx.findall('a=1, *DODGY*SYNTAX* b=2') [('a', '1'), ('b', '2')]

I'm going to save this one and study it, too. I'd like to learn
to use regexes better, even if I do try to avoid them when possible :)
 
R

R. David Murray

Paul McGuire said:
If you must cram all your code into a single source file, then
pyparsing would be problematic. But pyparsing's installation
footprint is really quite small, just a single Python source file. So
if your program spans more than one file, just add pyparsing.py into
the local directory along with everything else.

It isn't a matter of wanting to cram the code into a single source file.
I'm fixing a bug in a vendor-installed application. A ten line locally
maintained patch is bad enough, installing a whole new external dependency
is just Not An Option :)
 
T

Tim Chase

import re
s = """a=1,b="0234,)#($)@", k="7" """
rx = re.compile(r'[ ]*(\w+)=([^",]+|"[^"]*")[ ]*(?:,|$)')
rx.findall(s)
[('a', '1'), ('b', '"0234,)#($)@"'), ('k', '"7"')]
rx.findall('a=1, *DODGY*SYNTAX* b=2') [('a', '1'), ('b', '2')]

I'm going to save this one and study it, too. I'd like to learn
to use regexes better, even if I do try to avoid them when possible :)

This regexp is fairly close to the one I used, but I employed the
re.VERBOSE flag to split it out for readability. The above
breaks down as

[ ]* # optional whitespace, traditionally "\s*"
(\w+) # tag the variable name as one or more "word" chars
= # the literal equals sign
( # tag the value
[^",]+ # one or more non-[quote/comma] chars
| # or
"[^"]*" # quotes around a bunch of non-quote chars
) # end of the value being tagged
[ ]* # same as previously, optional whitespace ("\s*")
(?: # a non-capturing group (why?)
, # a literal comma
| # or
$ # the end-of-line/string
) # end of the non-capturing group

Hope this helps,

-tkc
 
J

John Machin

 >>> import re
 >>> s = """a=1,b="0234,)#($)@", k="7" """
 >>> rx = re.compile(r'[ ]*(\w+)=([^",]+|"[^"]*")[ ]*(?:,|$)')
 >>> rx.findall(s)
 [('a', '1'), ('b', '"0234,)#($)@"'), ('k', '"7"')]
 >>> rx.findall('a=1, *DODGY*SYNTAX* b=2')
 [('a', '1'), ('b', '2')]
I'm going to save this one and study it, too.  I'd like to learn
to use regexes better, even if I do try to avoid them when possible :)

This regexp is fairly close to the one I used, but I employed the
re.VERBOSE flag to split it out for readability.  The above
breaks down as

  [ ]*       # optional whitespace, traditionally "\s*"

No, it's optional space characters -- T'd regard any other type of
whitespace there as a stuff-up.
  (\w+)      # tag the variable name as one or more "word" chars
  =          # the literal equals sign
  (          # tag the value
  [^",]+     # one or more non-[quote/comma] chars
  |          # or
  "[^"]*"    # quotes around a bunch of non-quote chars
  )          # end of the value being tagged
  [ ]*       # same as previously, optional whitespace  ("\s*")

same correction as previously
  (?:        # a non-capturing group (why?)

a group because I couldn't be bothered thinking too hard about the
precedence of the | operator, and non-capturing because the OP didn't
want it captured.
  ,          # a literal comma
  |          # or
  $          # the end-of-line/string
  )          # end of the non-capturing group

Hope this helps,

Me too :)

Cheers,
John
 
P

Paul McGuire

 >>> import re
 >>> s = """a=1,b="0234,)#($)@", k="7" """
 >>> rx = re.compile(r'[ ]*(\w+)=([^",]+|"[^"]*")[ ]*(?:,|$)')
 >>> rx.findall(s)
 [('a', '1'), ('b', '"0234,)#($)@"'), ('k', '"7"')]
 >>> rx.findall('a=1, *DODGY*SYNTAX* b=2')
 [('a', '1'), ('b', '2')]
I'm going to save this one and study it, too.  I'd like to learn
to use regexes better, even if I do try to avoid them when possible :)

This regexp is fairly close to the one I used, but I employed the
re.VERBOSE flag to split it out for readability.  The above
breaks down as

  [ ]*       # optional whitespace, traditionally "\s*"
  (\w+)      # tag the variable name as one or more "word" chars
  =          # the literal equals sign
  (          # tag the value
  [^",]+     # one or more non-[quote/comma] chars
  |          # or
  "[^"]*"    # quotes around a bunch of non-quote chars
  )          # end of the value being tagged
  [ ]*       # same as previously, optional whitespace  ("\s*")
  (?:        # a non-capturing group (why?)
  ,          # a literal comma
  |          # or
  $          # the end-of-line/string
  )          # end of the non-capturing group

Hope this helps,

-tkc

Mightent there be whitespace on either side of the '=' sign? And if
you are using findall, why is the bit with the delimiting commas or
end of line/string necessary? I should think findall would just skip
over this stuff, like it skips over *DODGY*SYNTAX* in your example.

-- Paul
 
T

Tim Chase

Paul said:
import re
s = """a=1,b="0234,)#($)@", k="7" """
rx = re.compile(r'[ ]*(\w+)=([^",]+|"[^"]*")[ ]*(?:,|$)')
rx.findall(s)
[('a', '1'), ('b', '"0234,)#($)@"'), ('k', '"7"')]
rx.findall('a=1, *DODGY*SYNTAX* b=2')
[('a', '1'), ('b', '2')]
I'm going to save this one and study it, too. I'd like to learn
to use regexes better, even if I do try to avoid them when possible :)
This regexp is fairly close to the one I used, but I employed the
re.VERBOSE flag to split it out for readability. The above
breaks down as

[ ]* # optional whitespace, traditionally "\s*"
(\w+) # tag the variable name as one or more "word" chars
= # the literal equals sign
( # tag the value
[^",]+ # one or more non-[quote/comma] chars
| # or
"[^"]*" # quotes around a bunch of non-quote chars
) # end of the value being tagged
[ ]* # same as previously, optional whitespace ("\s*")
(?: # a non-capturing group (why?)
, # a literal comma
| # or
$ # the end-of-line/string
) # end of the non-capturing group

Mightent there be whitespace on either side of the '=' sign? And if
you are using findall, why is the bit with the delimiting commas or
end of line/string necessary? I should think findall would just skip
over this stuff, like it skips over *DODGY*SYNTAX* in your example.

Which would leave you with the solution(s) fairly close to what I
original posited ;-)

(my comment about the "non-capturing group (why?)" was in
relation to not needing to find the EOL/comma because findall()
doesn't need it, as Paul points out, not the precedence of the
"|" operator.)

-tkc
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,734
Messages
2,569,441
Members
44,832
Latest member
GlennSmall

Latest Threads

Top