Adjusting the 1024 byte stdin buffer limit

[brucoder]

Are there runtime settings that can be used to adjust the default 1024
byte stdin buffer limit or a buildtime setting in pyconfig.h? I have a
need to pump this up to permit input of a large data block via stdin.
Tim Jones

 

[Andrew Koenig]

Are there runtime settings that can be used to adjust the default 1024
byte stdin buffer limit or a buildtime setting in pyconfig.h? I have a
need to pump this up to permit input of a large data block via stdin.

What do you think you are being prevented from doing?

 

[brucoder]

Currently, when sending a data stream that exceeds 1024 bytes via
stdin, the stream blocks at the 1024th byte. This precludes completion
of the submission of the data stream.

Tim

 

[brucoder]

Currently, when sending a data stream that exceeds 1024 bytes via
stdin, the stream blocks at the 1024th byte. This precludes completion
of the submission of the data stream.

Tim

 

[Steve Holden]

Currently, when sending a data stream that exceeds 1024 bytes via
stdin, the stream blocks at the 1024th byte. This precludes completion
of the submission of the data stream.

But surely the blocking will only last as long as the consuming process
leaves buffered input unread?

regards
Steve

 

[Fredrik Lundh]

Currently, when sending a data stream that exceeds 1024 bytes via
stdin, the stream blocks at the 1024th byte. This precludes completion
of the submission of the data stream.

you can pass in a buffer size when you open a file:

class file(object)
| file(name[, mode[, buffering]]) -> file object
|
| Open a file. The mode can be 'r', 'w' or 'a' for reading (default),
| writing or appending. The file will be created if it doesn't exist
| when opened for writing or appending; it will be truncated when
| opened for writing. Add a 'b' to the mode for binary files.
| Add a '+' to the mode to allow simultaneous reading and writing.
| If the buffering argument is given, 0 means unbuffered, 1 means line
| buffered, and larger numbers specify the buffer size.
| Add a 'U' to mode to open the file for input with universal newline
| support. Any line ending in the input file will be seen as a '\n'
| in Python. Also, a file so opened gains the attribute 'newlines';
| the value for this attribute is one of None (no newline read yet),
| '\r', '\n', '\r\n' or a tuple containing all the newline types seen.
|
| 'U' cannot be combined with 'w' or '+' mode.
|
| Note: open() is an alias for file().

or use os.fdopen() to reopen an existing file handle:

fdopen(fd [, mode='r' [, bufsize]]) -> file_object

Return an open file object connected to a file descriptor.

assuming "sending via stdin" means using a pipe, this page explains why all
this probably won't matter:

http://www.opengroup.org/onlinepubs/7990989799/xsh/write.html

(see the "Write requests to a pipe or FIFO" section)

</F>

 

[brucoder]

Hi All,
-
Thanks, but it seems I may have been a bit more confused that even I
thought. It turns out to be an issue with Apple's OS X pty/tty
interface. Even though the mediator tool is set up to read in up to
20K of data, the pty interface blocks at 1024 bytes. When testing this
under Linux, I'm fine. And, Solaris 8 weaves an even more diabolical
255 character limit .
-
Looks like I'll need to be a bit more creative than simply spawning a
shell.
-
Tim

Currently, when sending a data stream that exceeds 1024 bytes via
stdin, the stream blocks at the 1024th byte. This precludes completion
of the submission of the data stream.

you can pass in a buffer size when you open a file:

class file(object)
| file(name[, mode[, buffering]]) -> file object
|
| Open a file. The mode can be 'r', 'w' or 'a' for reading (default),
| writing or appending. The file will be created if it doesn't exist
| when opened for writing or appending; it will be truncated when
| opened for writing. Add a 'b' to the mode for binary files.
| Add a '+' to the mode to allow simultaneous reading and writing.
| If the buffering argument is given, 0 means unbuffered, 1 means line
| buffered, and larger numbers specify the buffer size.
| Add a 'U' to mode to open the file for input with universal newline
| support. Any line ending in the input file will be seen as a '\n'
| in Python. Also, a file so opened gains the attribute 'newlines';
| the value for this attribute is one of None (no newline read yet),
| '\r', '\n', '\r\n' or a tuple containing all the newline types seen.
|
| 'U' cannot be combined with 'w' or '+' mode.
|
| Note: open() is an alias for file().

or use os.fdopen() to reopen an existing file handle:

fdopen(fd [, mode='r' [, bufsize]]) -> file_object

Return an open file object connected to a file descriptor.

assuming "sending via stdin" means using a pipe, this page explains why all
this probably won't matter:

http://www.opengroup.org/onlinepubs/7990989799/xsh/write.html

(see the "Write requests to a pipe or FIFO" section)

</F>