Multiple HTTP Sockets or Threads for file uploads.

[gberz3]

Hi All,

I have a client for whom I developed a small application using Ruby
and Curl to upload files to his file server (to and from Windows
boxes). He now wants that application to have multiple sockets in
order to upload more than one file at a time. I have no idea what
course is the safest here. Do I simply create threads that run the
"curl" command and let that be, or do I try a library such as (http://
www.chilkatsoft.com/upload-ruby.asp) and go from there? I'm honestly
not familiar with threading or chilkat.

Any suggestions would be most appreciated.

Regards.

 

[Robert Klemme]

I have a client for whom I developed a small application using Ruby
and Curl to upload files to his file server (to and from Windows
boxes). He now wants that application to have multiple sockets in
order to upload more than one file at a time. I have no idea what
course is the safest here. Do I simply create threads that run the
"curl" command and let that be,

You just need to fork multiple curl processes.

or do I try a library such as (http://
www.chilkatsoft.com/upload-ruby.asp) and go from there? I'm honestly
not familiar with threading or chilkat.

I do not know that lib. If it does provide multiple concurrent
uploads that's probably a good choice.

If you were doing it in pure Ruby (i.e. using net/http) you could use threads.

Kind regards

robert

 

[Zhukov Pavel]

you should use DRb with multi-process architecture since ruby uses
green-threads, that should be much faster.

 

[Robert Klemme]

you should use DRb with multi-process architecture since ruby uses
green-threads, that should be much faster.

This is not necessarily true for IO bound tasks. Also, if curl is used
you just need multiple child processes but not multiple threads.

Kind regards

robert

 

[gberz3]

This is not necessarily true for IO bound tasks. Also, if curl is used
you just need multiple child processes but not multiple threads.

Kind regards

robert

Alright, I'm currently using Threads and it has sped things up
tremendously. Thank you both for your suggestions.

One problem I'm having now is that sometimes a file will hang cURL,
and while the other subprocesses finish there might be one or two that
hang up. In this case I'd like to have some kind of granular control
over killing individual items without necessarily killing the whole
thread. I'd be afraid to kill the entire thread as it may not be
finished processing legitimate cURL processes when you send the kill
switch.

Thoughts?

 

[Robert Klemme]

Alright, I'm currently using Threads and it has sped things up
tremendously. Thank you both for your suggestions.

But note that the speedup likely comes from using multiple curl
processes - not from using multiple Ruby threads.

One problem I'm having now is that sometimes a file will hang cURL,
and while the other subprocesses finish there might be one or two that
hang up. In this case I'd like to have some kind of granular control
over killing individual items without necessarily killing the whole
thread. I'd be afraid to kill the entire thread as it may not be
finished processing legitimate cURL processes when you send the kill
switch.

Thoughts?

Killing threads does nothing to your external processes. You can either
kill curl and force its termination or you need to investigate whether
curl is built in a way that it responds to certain signals by dropping
the current upload.

Kind regards

robert

 

[Todd Fisher]

Hi All,

I have a client for whom I developed a small application using Ruby
and Curl to upload files to his file server (to and from Windows
boxes). He now wants that application to have multiple sockets in
order to upload more than one file at a time. I have no idea what
course is the safest here. Do I simply create threads that run the
"curl" command and let that be, or do I try a library such as (http://
www.chilkatsoft.com/upload-ruby.asp) and go from there? I'm honestly
not familiar with threading or chilkat.

Any suggestions would be most appreciated.

Regards.

You could try the curb library. This version included support for the
Multi interface, allowing multiple easy handles to perform concurrently.
Check it out: https://github.com/taf2/curb/tree

 

[Todd Fisher]

Alright, I'm currently using Threads and it has sped things up
tremendously. Thank you both for your suggestions.

One problem I'm having now is that sometimes a file will hang cURL,
and while the other subprocesses finish there might be one or two that
hang up. In this case I'd like to have some kind of granular control
over killing individual items without necessarily killing the whole
thread. I'd be afraid to kill the entire thread as it may not be
finished processing legitimate cURL processes when you send the kill
switch.

Thoughts?

This could be caused by a bad domain name causing a dns to take a long
time... if you install the c-ares library and recompile libcurl you can
at least select and get activity while dns is resolving...