"Dummy" IO object to push and pull data?

[Shay Hawkins]

Hello,

I'm relatively new to advanced Ruby programming, and I seem to be in
need of a "dummy" IO object that does nothing except allowing reading
and writing to it. (I suppose it could do more, but I'd like to keep it
as simple as possible.) What I mean is, I need to use a select statement
to check an array two IO objects that could have data to be read.
However, in other threads running, I will be needing to push data to
these objects.

I'm using Windows Vista, and what I've found out about IO.pipe doesn't
seem to work like I intended - at least, not on my operating system.

Here's an example of what I'm trying to do:

$objectA = ???
$objectB = ???

Thread.new() do
$objectA.puts("stuff")
sleep(10)
end

Thread.new() do
$objectB.puts("stuff")
sleep(15)
end

while(1)
ready = select([$objectA,$objectB])
next if(ready == nil)

for object in ready[0]
if(object == $objectA)
puts("10 milliseconds have passed in the first thread.")
puts(">> with message: #{object.gets()}")
elsif(object == $objectB)
puts("15 milliseconds have passed in the second thread.")
puts(">> with message: #{object.gets()}")
end
end
end

Does anyone know of a way I could accomplish this? If I'm not explaining
what I'm trying to do clearly enough, just ask what more you'd like to
know. Also, please don't comment on any of the conventions I use, unless
it directly pertains to the IO question - the above is an example, and
barely resembles the actual code.

 

[Ryan Davis]

I'm relatively new to advanced Ruby programming, and I seem to be in
need of a "dummy" IO object that does nothing except allowing reading
and writing to it. (I suppose it could do more, but I'd like to keep it
as simple as possible.) What I mean is, I need to use a select statement
to check an array two IO objects that could have data to be read.
However, in other threads running, I will be needing to push data to
these objects.

ri StringIO

 

[Shay Hawkins]

ri StringIO

Thanks, but I think I'm missing something:

test.rb:15:in 'select': can't convert StringIO to IO (TypeError)

objectA = StringIO.new("")
objectB = StringIO.new("")
...
ready = select([objectA,objectB])

What would be the alternative to using select on them?

I apologize if these are basic questions, but I've tried Google and
looked on both the IO and StringIO documentation pages with no luck. If
there's a better way for me to go about searching, do tell - I'm all for
nagging you guys less.

 

[andrew mcelroy]

Thanks, but I think I'm missing something:

Depending on your application, fakefs might be worth while.

Are you needing fake files?

Andrew McElroy

test.rb:15:in 'select': can't convert StringIO to IO (TypeError)

objectA = StringIO.new("")
objectB = StringIO.new("")
...
ready = select([objectA,objectB])

What would be the alternative to using select on them?

I apologize if these are basic questions, but I've tried Google and
looked on both the IO and StringIO documentation pages with no luck. If
there's a better way for me to go about searching, do tell - I'm all for
nagging you guys less.

 

[Shay Hawkins]

I've been using a workaround that seems to work fine, but certainly
isn't the best practice. It involves me creating a local TCPServer that
simply echos any lines it receives back to the socket, and creating a
bunch of TCPSockets. Then the sockets can use puts() in delayed threads
while they're all being checked in a loop { select() } to see if they
have data ready to be gets()'d.

 

[Robert Klemme]

ri StringIO

Thanks, but I think I'm missing something:

test.rb:15:in 'select': can't convert StringIO to IO (TypeError)

objectA = StringIO.new("")
objectB = StringIO.new("")
..
ready = select([objectA,objectB])

What would be the alternative to using select on them?

I apologize if these are basic questions, but I've tried Google and
looked on both the IO and StringIO documentation pages with no luck. If
there's a better way for me to go about searching, do tell - I'm all for
nagging you guys less.

If your scenario is such that for any particular thread you are always
either reading *or* writing then you should probably look at class Queue.

require 'thread'

io = Queue.new # well, this isn't an IO

writers = (1..5).map do
Thread.new do
rand(10).times do |i|
io.enq i
end
end
end

readers = (1..2).map do
Thread.new do
while (x=io.deq)
puts x
end
end
end

writers.each {|th| th.join}
readers.size.times {io.enq nil}
readers.each {|th| th.join}

Kind regards

robert

 

[Brian Candler]

ri StringIO

Thanks, but I think I'm missing something:

test.rb:15:in 'select': can't convert StringIO to IO (TypeError)

objectA = StringIO.new("")
objectB = StringIO.new("")
...
ready = select([objectA,objectB])

Ah, then you want something with a real underlying O/S file descriptor.

require 'socket'
a, b = Socket.pair(Socket::AF_UNIX, Socket::SOCK_STREAM, 0)

Then you can select on a while in another thread you write to b (or vice
versa)

 

[Brian Candler]

Oh, and there is also IO.pipe, which is unidirectional under most Unix
flavours.

 

[Shay Hawkins]

Oh, and there is also IO.pipe, which is unidirectional under most Unix
flavours.

I've tried pipes, but it seems to always think there's data to be read.
Here's the code I was testing it with:



pipeAA, pipeAB = IO.pipe()
pipeBA, pipeBB = IO.pipe()

Thread.new() do
loop do
sleep(1)
pipeAB.puts("1 sec")
end
end

Thread.new() do
loop do
sleep(5)
pipeBB.puts("5 sec")
end
end

loop do
# Should wait for one (or both) to have data, right?
ready = select([pipeAA,pipeBA])
next if(ready == nil)

# But it ALWAYS thinks there's data to be read...
for i in ready[0]
if(i == pipeAA)
puts("AA:")
puts(pipeAA.gets()) # ...meaning this locks the program up!
elsif(i == pipeBA)
puts("BA:")
puts(pipeBA.gets())
end
end
end



The output of that is a single line, "AA:", which is just before it
attempts to get the data from pipeAA. It doesn't continue, because it's
waiting for data from pipeAA, but it shouldn't have gotten to that step
if there wasn't data ready in the first place.

 

[Tony Arcieri]

[Note: parts of this message were removed to make it a legal post.]

Hello,

I'm relatively new to advanced Ruby programming, and I seem to be in
need of a "dummy" IO object that does nothing except allowing reading
and writing to it. (I suppose it could do more, but I'd like to keep it
as simple as possible.) What I mean is, I need to use a select statement
to check an array two IO objects that could have data to be read.
However, in other threads running, I will be needing to push data to
these objects.

If this is all just for thread synchronization, you shouldn't be using IO
primitives at all; you should be using thread synchronization primitives.

Try ri ConditionVariable and see if that accomplishes what you're trying to
do.

 

[Caleb Clausen]

I've tried pipes, but it seems to always think there's data to be read.
Here's the code I was testing it with:

My recollection is that on windows, select only works on sockets. It
doesn't do anything useful with pipes. You are far from being the
first programmer to regret that this is the case.

 

[Shay Hawkins]

My recollection is that on windows, select only works on sockets. It
doesn't do anything useful with pipes. You are far from being the
first programmer to regret that this is the case.

To make sure that I've been using it right, select is supposed to check
if the IO objects have data ready, and if so, return an array of the
ready objects?

require("socket")

server = TCPServer.new("localhost",0)
insock = TCPSocket.new(server.addr[3],server.addr[1])
outsock = server.accept()
server.close()

puts("Select starting...")
ready = select([outsock])
puts("Select is over.")

I created a quick test program to make sure that select was working how
I thought it did. When this program is run, it outputs "Select
starting...", and then locks up, because it's waiting for outsock to
receive data (which it never will). What I hoped it would do is output
"Select starting..." and then output "Select is over.", because no data
was ready to be read from outsock.

If I'm using it correctly, is there an alternative to this on Windows
machines?

 

[Caleb Clausen]

To make sure that I've been using it right, select is supposed to check
if the IO objects have data ready, and if so, return an array of the
ready objects?

It returns an _array_ of arrays of ready file handles.

require("socket")

server = TCPServer.new("localhost",0)
insock = TCPSocket.new(server.addr[3],server.addr[1])
outsock = server.accept()
server.close()

puts("Select starting...")
ready = select([outsock])
puts("Select is over.")

I created a quick test program to make sure that select was working how
I thought it did. When this program is run, it outputs "Select
starting...", and then locks up, because it's waiting for outsock to
receive data (which it never will). What I hoped it would do is output
"Select starting..." and then output "Select is over.", because no data
was ready to be read from outsock.

Right. Select never returns because nothing ever happens on any of the
file handles you asked it about. If you want a nonblocking select,
pass 0 for its timeout (final parameter). Even better, make some kind
of event for select to read. Either write some data to insock, or
close insock so the select there's some event to be read (even if just
end of file). (BTW, above snippet has unfortunate variable names;
insock and outsock should be swapped.)

 

[Tony Arcieri]

[Note: parts of this message were removed to make it a legal post.]

If I'm using it correctly, is there an alternative to this on Windows
machines?

I'd again ask if you're trying to do I/O or thread synchronization here. It
sure sounds like the latter to me.

ConditionVariables will work on Windows just fine.

 

[Shay Hawkins]

I'd again ask if you're trying to do I/O or thread synchronization here.
It
sure sounds like the latter to me.

ConditionVariables will work on Windows just fine.

Well, it seems the problem is primarily occuring with Open3.popen3
(which I downloaded the support for Windows from RAA
(http://raa.ruby-lang.org/project/win32-open3/)). When I am attempting
to call .gets() on the STDERR line, it won't let me input anything into
STDIN via .puts() until the .gets() method is over (they are being
called in seperate threads). I tried using select on STDERR so that it
wouldn't lock up with .gets() if there was no data, but as mentioned
earlier in this topic, select only works properly with sockets on
Windows machines.

If you think ConditionVariables would help me in this case, could you
please give me an example of how they would go about fixing this
problem?

 

[Brian Candler]

Well, it seems the problem is primarily occuring with Open3.popen3
(which I downloaded the support for Windows from RAA
(http://raa.ruby-lang.org/project/win32-open3/)). When I am attempting
to call .gets() on the STDERR line, it won't let me input anything into
STDIN via .puts() until the .gets() method is over (they are being
called in seperate threads).

Sorry, I've read that several times and I still don't understand. Can
you post a small example which demonstrates the problem, preferably
standalone?

You should have three IO objects from open3: one writable (connected to
stdin on the child), and two readable (connected to stdout and stderr on
the child).

That means you should be able to do

select([child_out, child_err])

to find out which, if any, has data available for reading. It will block
if neither is readable, unless you add a timeout:

select([child_out, child_err], nil, nil, 0)

If that's broken on Windows, your best option is to upgrade to a real
operating system

However, I believe you should also be able to read from child_out and
child_err in two separate Ruby threads, which may be the simplest way to
multiplex them. Try it and see:

Thread.new(child_out) do |fd|
while line = fd.gets
puts line
end
end
Thread.new(child_err) do |fd|
while line = fd.gets
puts line
end
end

I don't think condition variables will help you here, because you're
really talking over I/O to a separate process, whilst a ruby
ConditionVariable is something within a single process.

By the way, remember that gets will block until it gets a newline.
select() tells you that some data is available, but not how much, nor
whether there's a newline. So if the other process has sent a prompt
which doesn't end with newline, gets will hang.

You could try replacing xxx.gets with xxx.readpartial(1024) to read
whatever data is available.

 

[Robert Klemme]

If you think ConditionVariables would help me in this case, could you
please give me an example of how they would go about fixing this
problem?

I gave an example using Queue (also a tool for thread synchronization)
already. How about describing what problem you are trying to solve. So
far I have only read that there are some threads that want to write
something and other threads that want to read. For that you can
certainly use a Queue as demonstrated. What things do you want to
"write" and "read" and what do all your threads do otherwise (i.e. when
not reading or writing)?

Kind regards

robert

 

[Shay Hawkins]

Sorry, I've read that several times and I still don't understand. Can
you post a small example which demonstrates the problem, preferably
standalone?

My apologies: I'll see if I can elaborate a bit...

Well, I originally was trying to find a workaround to this, but it seems
the root of the problem is in the process' streams. I have a seperate
program that I did not create myself (and therefore cannot change the
code of) - this program puts output into stderr, and accepts input
through stdin.

I call Open3.popen3 to create the three streams. In a loop, I want to
constantly read input from stderr. (Assuming I assigned the pipes to
stdin, stdout, and stderr variable names.)

loop do
line = stderr.gets()
puts("Output: #{line}")
end

Then, if I receive an output of say, 0, I want to input something to the
program through stdin five seconds later.

loop do
line = stderr.gets().strip()
puts("Output: #{line}")

if(line == "0")
Thread.new() do
sleep(5)
stdin.puts("Received 0 five seconds ago.")
end
end
end

It creates the thread properly, but when the sleep expires and it is
ready to call stdin.puts(), it cannot do anything, because the loop
continued around and it is reading from stderr. However, when it
receives a new set of input from stderr, since the stream is momentarily
being unused, it can then grab the stdin stream and properly input the
old line.

As I mentioned before, I tried using select on stderr so that it
wouldn't tie up the stream, yet I've been told select only works
properly with sockets on Windows.

If that's broken on Windows, your best option is to upgrade to a real
operating system

Can't argue with you there

What things do you want to "write" and "read" and what do all your
threads do otherwise (i.e. when not reading or writing)?

See: the above example. If you need a different one, just ask.

 

[Shay Hawkins]

It creates the thread properly, but when the sleep expires and it is
ready to call stdin.puts(), it cannot do anything, because the loop
continued around and it is reading from stderr. However, when it
receives a new set of input from stderr, since the stream is momentarily
being unused, it can then grab the stdin stream and properly input the
old line.

Hm, let me give an example of what this would output. The process that
I'm creating the stream for outputs something every 10 seconds. As I
mentioned, if it outputs a 0, I want to give it an input of "Received 0
five seconds ago." five seconds later.

0:00:00 - Program started.
0:00:10 - Output: 3
0:00:20 - Output: 0
0:00:30 - Output: 2
0:00:30 - Input: Received 0 five seconds ago.

It's a rough example, yes, but the point is that stdin seems to wait
until stderr is free. I understand this may be normal, but is there an
alternative to select() that will prevent me from tying up the streams
if there's no ready data in stderr?

 

[Brian Candler]

I call Open3.popen3 to create the three streams. In a loop, I want to
constantly read input from stderr. (Assuming I assigned the pipes to
stdin, stdout, and stderr variable names.)

loop do
line = stderr.gets()
puts("Output: #{line}")
end

Then, if I receive an output of say, 0, I want to input something to the
program through stdin five seconds later.

loop do
line = stderr.gets().strip()
puts("Output: #{line}")

if(line == "0")
Thread.new() do
sleep(5)
stdin.puts("Received 0 five seconds ago.")
end
end
end

It creates the thread properly, but when the sleep expires and it is
ready to call stdin.puts(), it cannot do anything, because the loop
continued around and it is reading from stderr.

Perhaps it would be helpful to distinguish symptoms from your assumed
causes.

Firstly, are you saying that the stdin.puts line never completes? You
need to prove this. I suggest:

STDERR.puts "About to send..."
stdin.puts "Received 0"
STDERR.puts "Finished sending."

Secondly, if the puts never completes, are you sure that an exception is
not being raised within your thread? Such exceptions are usually silent,
unless you either join the thread (wait for it to complete), or you set

Thread.abort_on_exception = true

at the top of your program. I suggest you do the latter, to help rule
out a possible cause.

Thirdly, because of buffering, the stuff you wrote to stdin might not
actually be sent until you do

stdin.flush

Or you could do 'stdin.sync = true' after opening with popen3, so that
all writes to stdin are flushed automatically.

(So if you see "Finished sending." but the other program doesn't appear
to respond, that could be the underlying problem)

However, when it
receives a new set of input from stderr, since the stream is momentarily
being unused, it can then grab the stdin stream and properly input the
old line.

So prove that; e.g. the STDERR.puts "Finished sending." I suggested
before.

As I mentioned before, I tried using select on stderr so that it
wouldn't tie up the stream, yet I've been told select only works
properly with sockets on Windows.

No, if you are using threads, you do not need to do select.

Actually, Ruby will be using select() behind the scenes, and so if
Windows really is that broken, then you would never be able to run
multiple threads talking on different file descriptions; but I didn't
think it was quite that broken (you need a Windows expert though, I am
not one)

If the program really only outputs to stderr and nothing to stdout, then
it may be simpler just to redirect stderr to stdout, and then talk to it
using a normal IO.popen. That would be "yourcmd 2>&1" under Linux; I
don't know what the Windows equivalent would be.

Of course, if it weren't for the "5 second delay" requirement, then it
could be written quite happily without a thread:

while line = stderr.gets
line.chomp!
puts "Output: #{line}"
if line == "0"
stdin.puts "Sending something now"
stdin.flush
end
end