Simple question: combine a quoted string into a single token

[Squeamizh]

Hi,

I have a program which separates each line of a text file into tokens,
using whitespace as a delimiter (I do this with String.split). This
suits my needs for the most part, but now I need the ability to treat
quoted strings as single tokens. Note that the quoted string could be
multiple words, or even a 0-length string.

Could anyone recommend a basic strategy for doing this? Should I deal
with this when I first tokenize each line, or should I combine tokens
appropriately during parsing when I see a double-quote?

Help would be greatly appreciated.

 

[Kevin Olbrich]

Hi,

I have a program which separates each line of a text file into tokens,
using whitespace as a delimiter (I do this with String.split). This
suits my needs for the most part, but now I need the ability to treat
quoted strings as single tokens. Note that the quoted string could be
multiple words, or even a 0-length string.

Could anyone recommend a basic strategy for doing this? Should I deal
with this when I first tokenize each line, or should I combine tokens
appropriately during parsing when I see a double-quote?

Help would be greatly appreciated.

You could pull out all the quoted strings into an array and then delete
them from the original before processing it normally.


_Kevin
www.sciwerks.com

 

[Robert Klemme]

Hi,

I have a program which separates each line of a text file into tokens,
using whitespace as a delimiter (I do this with String.split). This
suits my needs for the most part, but now I need the ability to treat
quoted strings as single tokens. Note that the quoted string could be
multiple words, or even a 0-length string.

Could anyone recommend a basic strategy for doing this? Should I deal
with this when I first tokenize each line, or should I combine tokens
appropriately during parsing when I see a double-quote?

Help would be greatly appreciated.

Something along the lines of

line.scan %r{
"[^"]*" |
\S+
}x

HTH

robert