Re: sys call length limitation

Discussion in 'Perl Misc' started by Rainer Weikusat, Jan 22, 2014.

  1. * Tong * <> writes:
    > On Wed, 22 Jan 2014 11:38:35 +0000, Rainer Weikusat wrote:
    >
    >>> I just found out that there is a strict limitation on how many
    >>> characters one can stuff in between `` sys calls:
    >>>
    >>> $ echo "`cat HttpBody`" | wc -c 238566
    >>>
    >>> $ cat HttpBody | perl -e '$s = <>; print length $s; print `echo $s`'
    >>> 238566
    >>>
    >>> I.e., the echo works fine in my shell, but not OK within Perl.
    >>>
    >>> Any way to increase the sys call length limitation, ie the limitation
    >>> how many characters one can stuff in between ``?

    >>
    >> As George already pointed out, this limit usually limits the combined
    >> size of environment and command-line arguments.

    >
    > That's only partly true:


    {ARG_MAX}
    Maximum length of argument to the exec functions including environment data.
    Minimum Acceptable Value: {_POSIX_ARG_MAX}

    http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/limits.h.html

    NB: You may need to register before being allowed to access that.


    > $ getconf ARG_MAX
    > 2097152
    >
    > That's way bigger than the 238566 of my file. That's why the echo on the
    > shell doesn't fail.


    The reason the echo in the shell doesn't fail is most likely because
    echo is a shell-builtin, ie, there is no exec involved here. When I try
    this on my machine with a file of size 140,000,

    [rw@sable]/tmp#echo "`cat x`" | wc -c
    140001

    but

    [rw@sable]/tmp#/bin/echo "`cat x`" | wc -c
    bash: /bin/echo: Argument list too long
    0

    The perl-variant can deal with at most 131,067 bytes (getconf ARG_MAX
    returns the same value) and strace -f perl ... shows that

    [pid 2123] execve("/bin/sh", ["sh", "-c", "echo ;;;;;;;;;;;;;;;;;;;;;;;;;;;"...], [/* 26 vars */]) = -1 E2BIG (Argument list too long)

    see also

    "Limits on size of arguments and environment" section in the Linux
    execve(2) manpage, in particular,

    Additionally, the limit per string is 32 pages (the kernel
    constant MAX_ARG_STRLEN)
    Rainer Weikusat, Jan 22, 2014
    #1
    1. Advertising

  2. Rainer Weikusat

    Tim McDaniel Guest

    Re: [OT] sys call length limitation

    In article <n12Eu.321366$>,
    * Tong * <> wrote:
    >[Sorry that this is getting OT] because I'm thinking the easiest way
    >to solve my problem is just to raise the "ceiling" -- recompile my
    >kernel.


    It strikes me that that solution is like finding that it's too hard to
    push a pile of bricks down the road, and fixing it by getting more
    people to push, when you'd be much better advised to put them on a
    wagon. I suggest using temp file(s) and/or a pipe (at most one, to
    avoid deadlock, as already mentioned) to pass data to and from an
    external process.

    >On Wed, 22 Jan 2014 06:25:18 +0000, Tim McDaniel wrote:
    >> To be more precise, not OK within *your* Perl. On the system I'm
    >> on on the moment..,(it works)

    >
    >Hi Tim, did you compile your kernel yourself? How did you make it to
    >take such huge size of arguments?


    I didn't do anything. Panix, my ISP, the first UNIX-based ISP in New
    York City, has command-line shells for its users. Whether they run
    stock NetBSD 6.1.2 or have modified it, I don't know.

    --
    Tim McDaniel,
    Tim McDaniel, Jan 23, 2014
    #2
    1. Advertising

  3. Re: [OT] sys call length limitation

    On 2014-01-23 14:16, * Tong * <> wrote:
    > On Thu, 23 Jan 2014 06:02:25 +0000, Tim McDaniel wrote:
    >> It strikes me that that solution is like finding that it's too hard to
    >> push a pile of bricks down the road, and fixing it by getting more
    >> people to push, when you'd be much better advised to put them on a
    >> wagon. I suggest using temp file(s) and/or a pipe (at most one, to
    >> avoid deadlock, as already mentioned) to pass data to and from an
    >> external process.

    >
    > True, but that's beyond me, and even it is doable, it'll be more than 10x
    > complicated than merely:
    >
    > s,(x+),`echo $1 | wc -c`,eg
    >
    > From the maintenance point view, I think we have only one clear winner,


    I think we have a clear loser. This will break as soon as $1 contains
    some characters which have a special meaning for the shell. In your
    original posting you called the variable "httpbody" - this can of course
    be anything, but it seems likely that it's html or xml - which typically
    contains "<", ">", "&", ";" as well as single and/or double quotes,
    which makes quoting the stuff "interesting".

    > especially this is only a personal task -- KISS is paramount for me.


    IMHO writing a few lines more is a lot simpler than fixing this when it
    inevitably breaks.

    hp


    --
    _ | Peter J. Holzer | Fluch der elektronischen Textverarbeitung:
    |_|_) | | Man feilt solange an seinen Text um, bis
    | | | | die Satzbestandteile des Satzes nicht mehr
    __/ | http://www.hjp.at/ | zusammenpaƟt. -- Ralph Babel
    Peter J. Holzer, Jan 23, 2014
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Tim McDaniel

    Re: sys call length limitation

    Tim McDaniel, Jan 22, 2014, in forum: Perl Misc
    Replies:
    0
    Views:
    79
    Tim McDaniel
    Jan 22, 2014
  2. George Mpouras

    Re: sys call length limitation

    George Mpouras, Jan 22, 2014, in forum: Perl Misc
    Replies:
    0
    Views:
    66
    George Mpouras
    Jan 22, 2014
  3. Rainer Weikusat

    Re: sys call length limitation

    Rainer Weikusat, Jan 22, 2014, in forum: Perl Misc
    Replies:
    0
    Views:
    69
    Rainer Weikusat
    Jan 22, 2014
  4. Rainer Weikusat

    Re: [OT] sys call length limitation

    Rainer Weikusat, Jan 24, 2014, in forum: Perl Misc
    Replies:
    1
    Views:
    90
    Rainer Weikusat
    Jan 24, 2014
  5. gamo
    Replies:
    22
    Views:
    152
    Rainer Weikusat
    Jan 26, 2014
Loading...

Share This Page