Simulating Mouse Dragging

Discussion in 'Ruby' started by Erik Veenstra, Jul 29, 2005.

  1. Is it possible to simulate a mouse drag event in a browser (or
    any program in general) with Ruby? Something like:

    dragmouse(x1, y1, x2, y2)

    or

    dragmouse("windowname", x1, y1, x2, y2)

    Either Windows or Linux (Xfree86).

    Why? On the 'Net, there are a lot of sites which offer maps.
    You can scroll these maps by dragging the image, virtually
    flying over the map. You can look at such a map as one very,
    very big image, of which you see only one part at a time.
    Collecting these parts gives us little overlapping maps of a
    certain region. These overlapping maps can be converted to
    non-overlapping maps by cropping and saving with ImageMagick.
    The resulting "tiles" are zipped in a file, which can be
    handled with TIV (Tiled Image Viewer).

    This is no theory, it actually works. I've made TIV in Ewe [1]
    (kind of Java...). It runs on Windows (natively, fast), Linux
    (Java, slow) and on my iPAQ (natively, fast enough). I did the
    conversion from the overlapping maps to the non-overlapping
    maps with ImageMagick. Not difficult, but it takes some
    processing time. Saving the screenshot can be done with
    "import" (part of ImageMagick). I glued everything together
    with Ruby, of course.

    I once got the images by hacking the URL's of one of those
    sites, but they changed it... Grabbing the images by taking
    screenshots is much better, because it works for every
    "vendor"... The only missing part is the programmatically
    dragging of the images in the browser.

    Anybody able to help? Other ideas? (Except "buying" a
    product...)

    gegroet,
    Erik V. - http://www.erikveen.dds.nl/

    [1] http://www.ewesoft.com/
    Erik Veenstra, Jul 29, 2005
    #1
    1. Advertising

  2. Erik Veenstra

    Kero Guest

    > Is it possible to simulate a mouse drag event in a browser (or
    > any program in general) with Ruby? Something like:
    >
    > dragmouse(x1, y1, x2, y2)
    >
    > or
    >
    > dragmouse("windowname", x1, y1, x2, y2)
    >
    > Either Windows or Linux (Xfree86).


    Could recording + analyzing mouse data help? Then you can simulate it
    and let X11 consume it. `gpm -R` may help.

    +--- Kero ------------------------- kero@chello@nl ---+
    | all the meaningless and empty words I spoke |
    | Promises -- The Cranberries |
    +--- M38c --- http://members.chello.nl/k.vangelder ---+
    Kero, Jul 29, 2005
    #2
    1. Advertising

  3. On Fri, 29 Jul 2005 20:21:04 +0000, Kero wrote:

    > > Is it possible to simulate a mouse drag event in a browser
    > > (or any program in general) with Ruby? Something like:
    > >
    > > dragmouse(x1, y1, x2, y2)
    > >
    > > or
    > >
    > > dragmouse("windowname", x1, y1, x2, y2)
    > >
    > > Either Windows or Linux (Xfree86).

    >
    > Could recording + analyzing mouse data help? Then you can
    > simulate it and let X11 consume it. `gpm -R` may help.


    "Recording" the mouse data by reading /dev/gpmdata actually
    works. Good start. Generating such a data stream is possible.
    But what do we have to do to let X11 "consume" that stream?
    "cat test > /dev/gpmdata" doesn't work...

    gegroet,
    Erik V. - http://www.erikveen.dds.nl/
    Erik Veenstra, Jul 29, 2005
    #3
  4. Erik Veenstra

    Kero Guest

    >> > Is it possible to simulate a mouse drag event in a browser
    >> > (or any program in general) with Ruby? Something like:

    >>
    >> Could recording + analyzing mouse data help? Then you can
    >> simulate it and let X11 consume it. `gpm -R` may help.

    >
    > "Recording" the mouse data by reading /dev/gpmdata actually
    > works. Good start. Generating such a data stream is possible.
    > But what do we have to do to let X11 "consume" that stream?
    > "cat test > /dev/gpmdata" doesn't work...


    disclaimer: I have never tried anything like this myself.

    In what sense does it not work?
    (assuming you made X11 read /dev/gpmdata one way or the other).
    I would guess that `cat test` is "too fast".

    +--- Kero ------------------------- kero@chello@nl ---+
    | all the meaningless and empty words I spoke |
    | Promises -- The Cranberries |
    +--- M38c --- http://members.chello.nl/k.vangelder ---+
    Kero, Jul 30, 2005
    #4
  5. > > > > Is it possible to simulate a mouse drag event in a
    > > > > browser (or any program in general) with Ruby?
    > > > > Something like:
    > > >
    > > > Could recording + analyzing mouse data help? Then you can
    > > > simulate it and let X11 consume it. `gpm -R` may help.

    > >
    > > "Recording" the mouse data by reading /dev/gpmdata actually
    > > works. Good start. Generating such a data stream is
    > > possible. But what do we have to do to let X11 "consume"
    > > that stream? "cat test /dev/gpmdata" doesn't work...

    >
    > disclaimer: I have never tried anything like this myself.
    >
    > In what sense does it not work? (assuming you made X11 read
    > /dev/gpmdata one way or the other). I would guess that `cat
    > test` is "too fast".


    I experimented a bit, yesterday night. I didn't configure X11
    to read from /dev/gpmdata. But I found that one. After that,
    writing to /dev/gpmdata with a properly (?) filled memory
    structure, actually did move the mouse. Clicking was possible,
    too. But dragging somehow still doesn't work.

    The protocol is IMPS/2.

    I attached my experiment.

    Somebody able and willing to help?

    gegroet,
    Erik V. - http://www.erikveen.dds.nl/

    ----------------------------------------------------------------

    # http://www.mit.edu:8001/people/gassend/protocols/intellimouse/
    # http://www.microsoft.com/whdc/device/input/mcompat.mspx

    class Mouse
    Yov = 0x80
    Xov = 0x40
    Y8 = 0x20
    X8 = 0x10
    BIT4 = 0x08
    MB = 0x04
    RB = 0x02
    LB = 0x01

    def initialize
    @handle = File.new("/dev/gpmdata", "wb")
    @handle.sync = true
    @button = 0
    end

    def move(dx, dy)
    @bytes = []

    byte1 = (BIT4 | @button)
    byte2 = (dx.to_i)
    byte3 = (dy.to_i)
    byte4 = 0

    if dx < 0
    byte1 |= X8
    byte2 += 256
    end

    if dy < 0
    byte1 |= Y8
    byte3 += 256
    end

    @bytes << byte1
    @bytes << byte2
    @bytes << byte3
    @bytes << byte4

    send
    end

    def down(button)
    @button = button
    move(0, 0)
    end

    def up
    @button = 0
    move(0, 0)
    end

    def drag(dx, dy, button)
    down(button)
    move(dx, dy)
    up
    end

    def click(button)
    down(button)
    up
    end

    def stream
    @bytes.pack("c%d" % @bytes.length)
    end

    def inspect
    @bytes.collect{|s| s.chr}.join("").unpack("H2"*@bytes.length)
    end

    def send
    p inspect
    @handle << stream
    end
    end

    #system "sudo /etc/init.d/gpm stop" or puts "Oops"

    mouse = Mouse.new

    mouse.drag(-100, 0, Mouse::LB)
    mouse.move(+100, 0)

    #system "sudo /etc/init.d/gpm start" or puts "Oops"

    ----------------------------------------------------------------
    Erik Veenstra, Jul 30, 2005
    #5
  6. Am Samstag, den 30.07.2005, 05:01 +0900 schrieb Erik Veenstra:
    > Is it possible to simulate a mouse drag event in a browser (or
    > any program in general) with Ruby? Something like:
    >
    > dragmouse(x1, y1, x2, y2)
    >
    > or
    >
    > dragmouse("windowname", x1, y1, x2, y2)
    >
    > Either Windows or Linux (Xfree86).


    If you realy want to do it in a browser the simplest way would be to
    capture the MOUSEMOVE events in javascript.

    Another way it to build an app with Gtk+. There you have complete mouse
    controll and allso got image handling. The advantage over the browser
    solution is, that it would run on Linux, OS X and Windoze. Within
    javascript you have to program around the pittfalls of the different DOM
    models.

    Cheers
    detlef
    Detlef Reichl, Jul 30, 2005
    #6
  7. Erik Veenstra wrote:
    > Is it possible to simulate a mouse drag event in a browser (or
    > any program in general) with Ruby? Something like:
    >
    > dragmouse(x1, y1, x2, y2)
    >
    > or
    >
    > dragmouse("windowname", x1, y1, x2, y2)
    >
    > Either Windows or Linux (Xfree86).



    I've never used it, but AutoIt for windows may be able to do what
    you want: http://www.autoitscript.com/autoit3/index.php
    William James, Jul 30, 2005
    #7
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Davisro
    Replies:
    2
    Views:
    1,284
    Jim Corey
    Apr 23, 2004
  2. Wolfgang
    Replies:
    2
    Views:
    790
    S Manohar
    Feb 1, 2004
  3. Max
    Replies:
    7
    Views:
    9,072
  4. Pedro Sebastiao

    Simulating mouse clicks

    Pedro Sebastiao, May 24, 2004, in forum: C++
    Replies:
    2
    Views:
    1,732
    Prateek R Karandikar
    May 26, 2004
  5. Replies:
    0
    Views:
    242
Loading...

Share This Page