Threadpooling in Linux

Discussion in 'C Programming' started by Ram Prasad, Dec 1, 2011.

  1. Ram  Prasad

    Ram Prasad Guest

    Sorry for being OT
    I am trying to search for a simple C threadpool library for linux ,
    But almost all results lead to C# libraries.

    Is Threadpooling under linux not used or recommended ?

    I am trying to implement a multi-threaded app to send mails for a
    client. This has to connect to a mysql database get the records for
    the day and then send a mail.
    The current one in php is too slow , these are important market
    reports , which have to reach as soon as possible to the clients
    before the market opens for the day.

    I am trying to use a multi-threaded system so that I can maximize
    the throughput.

    What is the recommended way ?
    Ram Prasad, Dec 1, 2011
    #1
    1. Advertising

  2. Ram  Prasad

    Paul Guest

    Ram Prasad wrote:
    > Sorry for being OT
    > I am trying to search for a simple C threadpool library for linux ,
    > But almost all results lead to C# libraries.
    >
    > Is Threadpooling under linux not used or recommended ?
    >
    > I am trying to implement a multi-threaded app to send mails for a
    > client. This has to connect to a mysql database get the records for
    > the day and then send a mail.
    > The current one in php is too slow , these are important market
    > reports , which have to reach as soon as possible to the clients
    > before the market opens for the day.
    >
    > I am trying to use a multi-threaded system so that I can maximize
    > the throughput.
    >
    > What is the recommended way ?


    Have you studied the rate-limiting step in your architecture ?

    Where is the bottleneck ?

    (The email server has finite limits - profile the processing of
    each message, and see if all the time is spent conversing with
    the email server. Running in parallel might not help, because
    the email server will have a "number of connections" limit.)

    I have a feeling, you're assuming since you have a gigabit connection
    to the email server, it will run at one gigabit per second, if
    your program runs fast enough. The email server may only accept
    20 emails per second, and a dialup connection would have enough
    bandwidth for that. And if the email server is designed to
    accept messages faster, it only causes the internal queue in the
    email server to grow, and the messages won't leave the
    server for another eight hours.

    If the ISP or company had 100 email servers, you could fork 100
    threads and send 20 emails per second in each thread. Still no
    where near what you were expecting.

    You can write and run your own email server. Good luck with that.
    Who will accept a connection from you ? Will your server
    be instantly blacklisted ? Think of the upstream servers you
    will be overloading, with one million emails at 7:59AM in the
    morning. Now, those servers take eight hours to deliver
    all the mail.

    Start by benchmarking the processing rate of the email
    back end. If you really have infinite email capacity
    in your system, the rest of your project *will* be
    trivial.

    (An example of programmers fighting with email...)

    http://social.msdn.microsoft.com/Fo...m/thread/6d389dd4-8edf-49e8-883b-8787a0443610

    Paul
    Paul, Dec 1, 2011
    #2
    1. Advertising

  3. Ram  Prasad

    Ram Prasad Guest

    On Dec 1, 2:07 pm, Paul <> wrote:
    > Ram Prasad wrote:
    > > Sorry for being OT
    > > I am trying to search for a simple C threadpool library for linux ,
    > > But almost all results lead to C# libraries.

    >
    > > Is Threadpooling under linux not used or recommended ?

    >
    > > I am trying to implement a multi-threaded app to send mails for a
    > > client. This has to connect to a mysql database get the records for
    > > the day and then send a mail.
    > > The current one in php is too slow , these are important market
    > > reports , which have to reach as soon as possible to the clients
    > > before the market opens for the day.

    >
    > > I am trying to use a multi-threaded   system so that I can maximize
    > > the throughput.

    >
    > > What is the recommended way ?

    >
    > Have you studied the rate-limiting step in your architecture ?
    >
    > Where is the bottleneck ?
    >
    > (The email server has finite limits - profile the processing of
    > each message, and see if all the time is spent conversing with
    > the email server. Running in parallel might not help, because
    > the email server will have a "number of connections" limit.)
    >
    > I have a feeling, you're assuming since you have a gigabit connection
    > to the email server, it will run at one gigabit per second, if
    > your program runs fast enough. The email server may only accept
    > 20 emails per second, and a dialup connection would have enough
    > bandwidth for that. And if the email server is designed to
    > accept messages faster, it only causes the internal queue in the
    > email server to grow, and the messages won't leave the
    > server for another eight hours.
    >
    > If the ISP or company had 100 email servers, you could fork 100
    > threads and send 20 emails per second in each thread. Still no
    > where near what you were expecting.
    >
    > You can write and run your own email server. Good luck with that.
    > Who will accept a connection from you ? Will your server
    > be instantly blacklisted ? Think of the upstream servers you
    > will be overloading, with one million emails at 7:59AM in the
    > morning. Now, those servers take eight hours to deliver
    > all the mail.
    >
    > Start by benchmarking the processing rate of the email
    > back end. If you really have infinite email capacity
    > in your system, the rest of your project *will* be
    > trivial.
    >
    > (An example of programmers fighting with email...)
    >
    > http://social.msdn.microsoft.com/Forums/en-US/netfxnetcom/thread/6d38...
    >
    >     Paul


    In my case mail servers are our business. So if we can scale the Mail
    servers as much as required.

    The problem is not the SMTPD itself. I have written small testscripts
    and I am reasonably convinced that on commodity Dell hardware I am
    able to achieve 1M messages per hour provided delivered to a trusting
    network. ( No 4xx here :) )

    Even in the real world case these mails are delivered to people who
    have paid to receive these reports. Why would they get blacklisted for
    this ??

    The challenge is to send in parallel. Threading implementation is perl/
    php is not very popular AFAIK.Multi-processing is taking too much of I/
    O and bloating up the hardware requirement.

    That is precisely why I need a multi-threaded client.
    Ram Prasad, Dec 1, 2011
    #3
  4. Ram  Prasad

    Noob Guest

    Ram Prasad wrote:

    > Sorry for being OT


    Since you know your question is off-topic, why don't you post
    in more appropriate a forum?

    Here are a few suggestions.

    comp.unix.programmer
    comp.os.linux.development.apps
    comp.programming.threads

    Regards.
    Noob, Dec 1, 2011
    #4
  5. Ram  Prasad

    Jorgen Grahn Guest

    On Thu, 2011-12-01, Ram Prasad wrote:
    > On Dec 1, 2:07 pm, Paul <> wrote:
    >> Ram Prasad wrote:
    >> > Sorry for being OT
    >> > I am trying to search for a simple C threadpool library for linux ,
    >> > But almost all results lead to C# libraries.

    >>
    >> > Is Threadpooling under linux not used or recommended ?

    >>
    >> > I am trying to implement a multi-threaded app to send mails for a
    >> > client. This has to connect to a mysql database get the records for
    >> > the day and then send a mail.
    >> > The current one in php is too slow , these are important market
    >> > reports , which have to reach as soon as possible to the clients
    >> > before the market opens for the day.

    >>
    >> > I am trying to use a multi-threaded   system so that I can maximize
    >> > the throughput.

    >>
    >> > What is the recommended way ?

    >>
    >> Have you studied the rate-limiting step in your architecture ?
    >>
    >> Where is the bottleneck ?
    >>

    ....

    > In my case mail servers are our business. So if we can scale the Mail
    > servers as much as required.


    That last sentence is not possible to understand -- what did you mean
    to say?

    > The problem is not the SMTPD itself. I have written small testscripts
    > and I am reasonably convinced that on commodity Dell hardware I am
    > able to achieve 1M messages per hour provided delivered to a trusting
    > network. ( No 4xx here :) )

    ....
    > The challenge is to send in parallel. Threading implementation is perl/
    > php is not very popular AFAIK.Multi-processing is taking too much of I/
    > O and bloating up the hardware requirement.
    >
    > That is precisely why I need a multi-threaded client.


    I still don't understand why you think you need that.

    First, /usr/lib/sendmail can send the same mail to thousands of
    recipients. The local MTA will take the hit and do something sensible,
    (like forwarding to all at example.com in one session).

    But perhaps you feel you need to personalize the mails.

    Second, internet mail is store-and-forward. If you let a thousand
    /usr/lib/sendmail processes hit the same local MTA concurrently, the
    job each will be waiting for is just an acknowledgement that the mail
    has been locally enqueued. That's an I/O-bound operation, and
    threading is of little use.

    Are you perhaps sidestepping /usr/lib/sendmail, trying to do its job?

    /Jorgen

    --
    // Jorgen Grahn <grahn@ Oo o. . .
    \X/ snipabacken.se> O o .
    Jorgen Grahn, Dec 4, 2011
    #5
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Tim Van Holder

    Re: CLR for Linux??

    Tim Van Holder, Jun 29, 2003, in forum: ASP .Net
    Replies:
    0
    Views:
    4,331
    Tim Van Holder
    Jun 29, 2003
  2. gnu

    Why not use Linux?

    gnu, Nov 14, 2003, in forum: ASP .Net
    Replies:
    18
    Views:
    900
    Giacomo
    Nov 22, 2003
  3. U. George
    Replies:
    4
    Views:
    835
    U. George
    Aug 11, 2005
  4. Daniel Bush
    Replies:
    8
    Views:
    128
    Tony Arcieri
    Oct 21, 2009
  5. Krishna Chaitanya
    Replies:
    1
    Views:
    138
    Krishna Chaitanya
    Feb 27, 2009
Loading...

Share This Page