Threadpooling in Linux

[Ram Prasad]

Sorry for being OT
I am trying to search for a simple C threadpool library for linux ,
But almost all results lead to C# libraries.

Is Threadpooling under linux not used or recommended ?

I am trying to implement a multi-threaded app to send mails for a
client. This has to connect to a mysql database get the records for
the day and then send a mail.
The current one in php is too slow , these are important market
reports , which have to reach as soon as possible to the clients
before the market opens for the day.

I am trying to use a multi-threaded system so that I can maximize
the throughput.

What is the recommended way ?

 

[Paul]

Sorry for being OT
I am trying to search for a simple C threadpool library for linux ,
But almost all results lead to C# libraries.

Is Threadpooling under linux not used or recommended ?

I am trying to implement a multi-threaded app to send mails for a
client. This has to connect to a mysql database get the records for
the day and then send a mail.
The current one in php is too slow , these are important market
reports , which have to reach as soon as possible to the clients
before the market opens for the day.

I am trying to use a multi-threaded system so that I can maximize
the throughput.

What is the recommended way ?

Have you studied the rate-limiting step in your architecture ?

Where is the bottleneck ?

(The email server has finite limits - profile the processing of
each message, and see if all the time is spent conversing with
the email server. Running in parallel might not help, because
the email server will have a "number of connections" limit.)

I have a feeling, you're assuming since you have a gigabit connection
to the email server, it will run at one gigabit per second, if
your program runs fast enough. The email server may only accept
20 emails per second, and a dialup connection would have enough
bandwidth for that. And if the email server is designed to
accept messages faster, it only causes the internal queue in the
email server to grow, and the messages won't leave the
server for another eight hours.

If the ISP or company had 100 email servers, you could fork 100
threads and send 20 emails per second in each thread. Still no
where near what you were expecting.

You can write and run your own email server. Good luck with that.
Who will accept a connection from you ? Will your server
be instantly blacklisted ? Think of the upstream servers you
will be overloading, with one million emails at 7:59AM in the
morning. Now, those servers take eight hours to deliver
all the mail.

Start by benchmarking the processing rate of the email
back end. If you really have infinite email capacity
in your system, the rest of your project *will* be
trivial.

(An example of programmers fighting with email...)

http://social.msdn.microsoft.com/Fo...m/thread/6d389dd4-8edf-49e8-883b-8787a0443610

Paul

 

[Ram Prasad]

Have you studied the rate-limiting step in your architecture ?

Where is the bottleneck ?

(The email server has finite limits - profile the processing of
each message, and see if all the time is spent conversing with
the email server. Running in parallel might not help, because
the email server will have a "number of connections" limit.)

I have a feeling, you're assuming since you have a gigabit connection
to the email server, it will run at one gigabit per second, if
your program runs fast enough. The email server may only accept
20 emails per second, and a dialup connection would have enough
bandwidth for that. And if the email server is designed to
accept messages faster, it only causes the internal queue in the
email server to grow, and the messages won't leave the
server for another eight hours.

If the ISP or company had 100 email servers, you could fork 100
threads and send 20 emails per second in each thread. Still no
where near what you were expecting.

You can write and run your own email server. Good luck with that.
Who will accept a connection from you ? Will your server
be instantly blacklisted ? Think of the upstream servers you
will be overloading, with one million emails at 7:59AM in the
morning. Now, those servers take eight hours to deliver
all the mail.

Start by benchmarking the processing rate of the email
back end. If you really have infinite email capacity
in your system, the rest of your project *will* be
trivial.

(An example of programmers fighting with email...)

http://social.msdn.microsoft.com/Forums/en-US/netfxnetcom/thread/6d38...

    Paul

In my case mail servers are our business. So if we can scale the Mail
servers as much as required.

The problem is not the SMTPD itself. I have written small testscripts
and I am reasonably convinced that on commodity Dell hardware I am
able to achieve 1M messages per hour provided delivered to a trusting
network. ( No 4xx here )

Even in the real world case these mails are delivered to people who
have paid to receive these reports. Why would they get blacklisted for
this ??

The challenge is to send in parallel. Threading implementation is perl/
php is not very popular AFAIK.Multi-processing is taking too much of I/
O and bloating up the hardware requirement.

That is precisely why I need a multi-threaded client.

 

[Noob]

Sorry for being OT

Since you know your question is off-topic, why don't you post
in more appropriate a forum?

Here are a few suggestions.

comp.unix.programmer
comp.os.linux.development.apps
comp.programming.threads

Regards.

 

[Jorgen Grahn]

....

In my case mail servers are our business. So if we can scale the Mail
servers as much as required.

That last sentence is not possible to understand -- what did you mean
to say?

The problem is not the SMTPD itself. I have written small testscripts
and I am reasonably convinced that on commodity Dell hardware I am
able to achieve 1M messages per hour provided delivered to a trusting
network. ( No 4xx here ) ....
The challenge is to send in parallel. Threading implementation is perl/
php is not very popular AFAIK.Multi-processing is taking too much of I/
O and bloating up the hardware requirement.

That is precisely why I need a multi-threaded client.

I still don't understand why you think you need that.

First, /usr/lib/sendmail can send the same mail to thousands of
recipients. The local MTA will take the hit and do something sensible,
(like forwarding to all at example.com in one session).

But perhaps you feel you need to personalize the mails.

Second, internet mail is store-and-forward. If you let a thousand
/usr/lib/sendmail processes hit the same local MTA concurrently, the
job each will be waiting for is just an acknowledgement that the mail
has been locally enqueued. That's an I/O-bound operation, and
threading is of little use.

Are you perhaps sidestepping /usr/lib/sendmail, trying to do its job?

/Jorgen