Processing std::strings into data

[GGG]

I have a situation where at tool is passing me a large array of
strings that I need to process in a particular type of data. Each item
in the array gets to me as a pair of std::strings, basically, one
string represents the data, one string represents the data type.
Here's a quick example of what it sort of looks like

string data[4] = { "0.0000002342", "12.00234", "42", "5" };
string types[4] = { "EF", "EF", "I", "I" }; //EF is 64 bit float,
I is 32 bit int

So element 0 of data matches with element 0 of types. i.e. types[0]
defines what type of data data[0] is.

This example is simple, however speed is going to be crucial as I need
to process a thousands of these in very short order. So I am looking
at ways make this work as fast as possible. I can't change the way the
data is being passed to me. I'm stuck with 2 std:string arrays. I'm
thinking a stringstream can work simple for this, but on my current
tests, my target machine can barley process this fast enough...

So I thought I could do something like this(using my dummy data above)

double doubleVal;
int intVal;

for( int i = 0; i < 4; i++) //just hardcode 4 as the size for this
example
{
stringstream tmpStream(data);
if ( types == "EF" )
{
tmpStream >> doubleVal;
//process the double here....
}
else if ( types == "I" )
{
tmpStream >> intVal;
//process the integer here....
}
else
{
cerr << "kerblooy" << endl;
}
}

So basically I am wondering if there is something faster here than
using stringstream, or... is there a way I can declare stringstream
outside of the for loop so its destructor/constructor are not being
called constantly?

Now... I am already working on optimizing the "//process the value
here" part. I've played around with the code an seem to have found
that just this looping part is causing me a little more than have my
CPU usage time.

Any ideas would be cool.
-grant

 

[Leor Zolman]

I have a situation where at tool is passing me a large array of
strings that I need to process in a particular type of data. Each item
in the array gets to me as a pair of std::strings, basically, one
string represents the data, one string represents the data type.
Here's a quick example of what it sort of looks like

string data[4] = { "0.0000002342", "12.00234", "42", "5" };
string types[4] = { "EF", "EF", "I", "I" }; //EF is 64 bit float,
I is 32 bit int

So element 0 of data matches with element 0 of types. i.e. types[0]
defines what type of data data[0] is.

This example is simple, however speed is going to be crucial as I need
to process a thousands of these in very short order. So I am looking
at ways make this work as fast as possible. I can't change the way the
data is being passed to me. I'm stuck with 2 std:string arrays. I'm
thinking a stringstream can work simple for this, but on my current
tests, my target machine can barley process this fast enough...

So I thought I could do something like this(using my dummy data above)

double doubleVal;
int intVal;

for( int i = 0; i < 4; i++) //just hardcode 4 as the size for this
example
{
stringstream tmpStream(data);
if ( types == "EF" )
{
tmpStream >> doubleVal;
//process the double here....
}
else if ( types == "I" )
{
tmpStream >> intVal;
//process the integer here....
}
else
{
cerr << "kerblooy" << endl;
}
}

So basically I am wondering if there is something faster here than
using stringstream, or... is there a way I can declare stringstream
outside of the for loop so its destructor/constructor are not being
called constantly?

Now... I am already working on optimizing the "//process the value
here" part. I've played around with the code an seem to have found
that just this looping part is causing me a little more than have my
CPU usage time.

Any ideas would be cool.

I think you can do better, but precisely what approach you take may be
dependent on some details you haven't shown. For example, how many
different "types" are there? If the number is small, an if...else core to
your loop may be optimal. If the number is large, pre-loading a
map<string,int> with the types as keys and using the value (the int) as the
dispatch value in a switch may work better.

As for the string-to-value conversion, avoiding the constructor call for
the stringstream (it might be faster to use an istringstream, but I'm not
sure) would depend on finding a way to "reset" the istringstream with a new
string without actually re-initializing the whole thing, and I'm not quite
sure if there's a consensus on whether or not there's a way to even do that
(elegant or otherwise). But even if there is, don't forget about good ole'
sscanf, along with specialized library functions for the different types
(strtoul, atof, etc.). Using these would probably make a reasonable-sized
dent.

HTH,
-leor

 

[Jerry Coffin]

(e-mail address removed) (GGG) wrote in message
[ ... this works, but it's too slow: ]

double doubleVal;
int intVal;

for( int i = 0; i < 4; i++) //just hardcode 4 as the size for this
example
{
stringstream tmpStream(data);
if ( types == "EF" )
{
tmpStream >> doubleVal;
//process the double here....
}
else if ( types == "I" )
{
tmpStream >> intVal;
//process the integer here....
}
else
{
cerr << "kerblooy" << endl;
}
}

Here's a test program of a possible alternative:

#include <sstream>
#include <cstdlib>
#include <iostream>
#include <string>
#include <ctime>

std::string data[4] = { "0.0000002342", "12.00234", "42", "5" };
std::string types[4] = { "EF", "EF", "I", "I" }; //EF is 64 bit
float,

int ints = 0;
int doubles = 0;

void process(int i) {
ints ++;
}

void process(double d) {
doubles++;
}

void convert1() {
double doubleVal;
int intVal;

for( int i = 0; i < 4; i++) //just hardcode 4 as the size for
this example
{
if ( types == "EF" )
{
doubleVal = std::atof(data.c_str());
process(doubleVal);
}
else if ( types == "I" )
{
intVal = std::atoi(data.c_str());
process(intVal);
}
else
{
std::cerr << "kerblooy" << std::endl;
}
}
}

void convert2() {
double doubleVal;
int intVal;

for( int i = 0; i < 4; i++) //just hardcode 4 as the size for this
example
{
std::stringstream tmpStream(data);
if ( types == "EF" )
{
tmpStream >> doubleVal;
process(doubleVal);
}
else if ( types == "I" )
{
tmpStream >> intVal;
process(intVal);
}
else
{
std::cerr << "kerblooy" << std::endl;
}
}
}

void show_time(std::string caption, void (*f)()) {
std::clock_t start = std::clock();
for (int i=0; i<100000; ++i)
f();
std::clock_t end = std::clock();

std::cerr << "Time " << caption << " " <<
double(end-start)/CLOCKS_PER_SEC << " seconds\n";
}

int main() {
show_time("using atoi/atof", convert1);
show_time("using stringstream", convert2);
std::cout << "int breaker: " << ints << std::endl;
std::cout << "double breaker: " << doubles << std::endl;
return 0;
}

Differences in speed obviously depend on the compiler and optimization
used, but with the compilers and libraries I have handy, the atoi/atof
version is always at least 3 times as fast, and sometimes up to 7
times as fast.

The calls to process and printing out the "breaker" numbers is to
break optimizers -- a good optimizer can eliminate the convesions if
it finds that the results are never used, so we ensure that they're
used (in a way that should have minimal effect on overall speed).

 

[GGG]

I think you can do better, but precisely what approach you take may be
dependent on some details you haven't shown. For example, how many
different "types" are there? If the number is small, an if...else core to
your loop may be optimal. If the number is large, pre-loading a
map<string,int> with the types as keys and using the value (the int) as the
dispatch value in a switch may work better.

As for the string-to-value conversion, avoiding the constructor call for
the stringstream (it might be faster to use an istringstream, but I'm not
sure) would depend on finding a way to "reset" the istringstream with a new
string without actually re-initializing the whole thing, and I'm not quite
sure if there's a consensus on whether or not there's a way to even do that
(elegant or otherwise). But even if there is, don't forget about good ole'
sscanf, along with specialized library functions for the different types
(strtoul, atof, etc.). Using these would probably make a reasonable-sized
dent.

HTH,
-leor

Well, I can keep the if/else, there are a total of 5 types(much to my
happy releif... this particular software has been known to be...
bloaty...)

Wow... I made a test app to compare the speed of a few of these
methods... it basically was a 200000 cylce for loop that ran each of
the following methods( stringstream, istringstream, and scanf)

The average results(no compiler optimizations)
stringstream: 3.53 secs
istringstream: 3.15 secs
sscanf: 0.62 secs

average results WITH opt
stringstream: 3.37 secs
istringstream: 3.05 secs
sscanf: 0.62 secs


So it looks like optimizations help a tiny bit for the stringstream,
istringstream seems faster than stringstream, but scanf is the hands
down winner. Makes sense I know... I was a bit suprised however at the
that magnitude of difference.


Groovy
-grant