multithreading sysread error

[Venkat Bagam]

Hi All,

I am a newbie at scraping and multi-threading too. Recently, I
have to implement these two in one of my application. Please find my
code in the attachment. I was facing some peculiar issues and the reason
I don't know!!! The code works pretty well on my ubuntu7.04 but throws
some "sysread", "read_status_line" errors when run on Windows. What
could be the obvious problem? Any help greatly appreciated.

regards,
Venkat Bagam

Attachments:
http://www.ruby-forum.com/attachment/1517/scrape.rb

 

[MenTaLguY]

There are a number of problems, among them:

1. deriving from Monitor does not do anything for you in
this case, since you don't use any of its locking
facilities (deriving from Monitor like this probably
isn't a good approach anyway)

2. WWW::Mechanize instances are not safe to share
between threads; it's best to create a separate agent
per thread.

3. Using the Timeout class on complicated libraries
can often break them. If you really need operations
to time out, it is best to see if the library provides
direct support for timeouts on its operations.

-mental

 

[Daniel Brumbaugh Keeney]

Hi All,

I am a newbie at scraping and multi-threading too. Recently, I
have to implement these two in one of my application. Please find my
code in the attachment. I was facing some peculiar issues and the reason
I don't know!!! The code works pretty well on my ubuntu7.04 but throws
some "sysread", "read_status_line" errors when run on Windows. What
could be the obvious problem? Any help greatly appreciated.

regards,
Venkat Bagam

sysread uses a "low-level" read. In general, I wouldn't be confident
that anything marked as "low-level" can be mixed well with
multi-threading. On my system, sysread is slower than ordinary read,
so I fail to see the advantage versus the standard IO#read.

Daniel Brumbaugh Keeney

 

[Venkat Bagam]

There are a number of problems, among them:

1. deriving from Monitor does not do anything for you in
this case, since you don't use any of its locking
facilities (deriving from Monitor like this probably
isn't a good approach anyway)

-mental

I used a synchronized block in "download_file(html_link)"
method...Doesn't that make any sense of locking ??? Am I wrong again !!!