Running windows ruby, and watir under wine on linux?

[Ezra Zygmuntowicz]

Friends-

I have seen a few blog posts where folks mention that they have
gotten the ruby one click installer and watir/IE running under wine
on linux. Has anyone out there done this successfully? My company
would be willing to pay someone to help us get this up and running.
Anyone have any tips or any more information about making this kind
of thing happen?

We are using watir as a web scraper tool of sorts for scraping
crusty sites that use multiple framesets and java applets and all
kinds of other horrible stuff that defies any type of regular screen
scraping short of using watir to actually navigate the sites and
enter data into forms to get results we need from these companies
sites. And we need multiple instances of watir driving the same
companies site at once which is problematic because internet explorer
overwrites its cookies when more then one of these scrapes is run
against the same website at once. So we are trying to experiment with
running watir under wine so each scaper runs in its own env and
doesn't step on each others cookies.

Any win32 gurus have any ideas as to how we might be able to
dynamically from our ruby/watir scripts, manage the cookies written
by IE so we could control where they get written and be able to run
two or more concurrent scrapes of the same website via watir without
IE overwriting the cookie it uses? My company is willing to pay
someone to help solve this problem. Thoughts or ideas are very
welcome at this point. I have asked this same thing eslewhere but I
know there are many smart folks on this list and I hope someone might
have an idea to help us out. More info available if you need it.


Thanks-
-Ezra

 

[Sean O'Halpin]

Friends-

I have seen a few blog posts where folks mention that they have
gotten the ruby one click installer and watir/IE running under wine
on linux. Has anyone out there done this successfully? My company
would be willing to pay someone to help us get this up and running.
Anyone have any tips or any more information about making this kind
of thing happen?

We are using watir as a web scraper tool of sorts for scraping
crusty sites that use multiple framesets and java applets and all
kinds of other horrible stuff that defies any type of regular screen
scraping short of using watir to actually navigate the sites and
enter data into forms to get results we need from these companies
sites. And we need multiple instances of watir driving the same
companies site at once which is problematic because internet explorer
overwrites its cookies when more then one of these scrapes is run
against the same website at once. So we are trying to experiment with
running watir under wine so each scaper runs in its own env and
doesn't step on each others cookies.

Any win32 gurus have any ideas as to how we might be able to
dynamically from our ruby/watir scripts, manage the cookies written
by IE so we could control where they get written and be able to run
two or more concurrent scrapes of the same website via watir without
IE overwriting the cookie it uses? My company is willing to pay
someone to help solve this problem. Thoughts or ideas are very
welcome at this point. I have asked this same thing eslewhere but I
know there are many smart folks on this list and I hope someone might
have an idea to help us out. More info available if you need it.


Thanks-
-Ezra

You could try runas
(http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/runas.mspx)
to run multiple Watir (and hence automated IE) processes as different
users, which would put the cookies in separates profiles.

Wouldn't work if you're trying to run multiple IE processes via
separate Ruby threads though.

Regards,
Sean

 

[Ezra Zygmuntowicz]

You could try runas
(http://www.microsoft.com/resources/documentation/windows/xp/all/
proddocs/en-us/runas.mspx)
to run multiple Watir (and hence automated IE) processes as different
users, which would put the cookies in separates profiles.

Wouldn't work if you're trying to run multiple IE processes via
separate Ruby threads though.

Regards,
Sean

Thanks Sean. We are already using runas I should have mentioned. THe
reason this is failing is because some of these sites have modal
dialogs and the only way to deal with these in watir is via
click_no_wait. What click no wait does is uses system to start
another process so the modal dialog doesn't block the entire ruby
process. THen when the separate process tries to re-attach to the
window there is a security error that won't let a different process
then the one that started ie bind to the window. This all goes away
if we can get a way to dynamically manage the cookies somehow. Or
under wine each instance if wine will have its own environment that
skirts the issue as well. We have been up and down the wall with this
issue for the past week and have come up with no answers yet. I do
appreciate the suggestion though.

Thanks
-Ezra

 

[Daniel Berger]

Thanks Sean. We are already using runas I should have mentioned. THe
reason this is failing is because some of these sites have modal
dialogs and the only way to deal with these in watir is via
click_no_wait. What click no wait does is uses system to start another
process so the modal dialog doesn't block the entire ruby process.
THen when the separate process tries to re-attach to the window there
is a security error that won't let a different process then the one
that started ie bind to the window. This all goes away if we can get a
way to dynamically manage the cookies somehow. Or under wine each
instance if wine will have its own environment that skirts the issue
as well. We have been up and down the wall with this issue for the
past week and have come up with no answers yet. I do appreciate the
suggestion though.

Thanks
-Ezra

Rather than using system() you could try using Process.create in the
win32-process package. I'm guessing somewhat here, but give this a shot:

require 'win32/process'
include Win32

def click_no_wait
Process.create(
:app_name => "c:\ruby\bin\ruby ...",
:creation_flags => Process::CREATE_PRESERVE_CODE_AUTHZ_LEVEL
:inherited => true
)
end

You can futz around with the various creation flags and see if any of
them help you out.

HTH,

Dan

 

[Ezra Zygmuntowicz]

Hi-

Rather than using system() you could try using Process.create in
the win32-process package. I'm guessing somewhat here, but give
this a shot:

require 'win32/process'
include Win32

def click_no_wait
Process.create(
:app_name => "c:\ruby\bin\ruby ...",
:creation_flags => Process::CREATE_PRESERVE_CODE_AUTHZ_LEVEL
:inherited => true
)
end

You can futz around with the various creation flags and see if any
of them help you out.

HTH,

Dan

Hey Dan-

Thanks a lot. That points us in the right direction. I really
appreciate it.

Thanks
-Ezra