Unix shell script with Perl one-liner causes error

Discussion in 'Perl Misc' started by Simon O, Jun 15, 2006.

  1. Simon O

    Simon O Guest

    Hi there

    I have this Unix Borne shell script that does something like this in a
    loop :

    for FILE in *
    do
    ROW_COUNT_HEADER=`...`
    if [ ${ROW_COUNT_HEADER} -ne 0 ]
    then
    ...
    # 1) OLD Option with Sed :
    # cat ${FILE} | cut -f1-32 | sed "s/$/ ${FILE}/g" >> file.dat
    # 2) NEW Option with Perl :
    cat ${FILE} | cut -f1-32 | perl -ple "s/$/\t${FILE}/g" >>
    file.dat
    fi
    bzip2 ${FILE}
    mv ${FILE}.bz2 ${LATEST_ARCHIVE}/IGCPDATA/
    done

    I was having problems with using sed, so changed to option 2) using
    perl which then gave a really weird error. It does what it's supposed
    to but when it gets to the bzip / move section, it gives an error as
    though it's still busy with ${FILE} ?? - being driving me MAD ! Almost
    as though Perl is has said it's finished but is not ?

    The error is this :

    bzip2: I/O or other error, bailing out. Possible reason follows.
    bzip2: No such file or directory
    Input file = ABS_MTM_By_Trade, output file =
    ABS_MTM_By_Trade.bz2
    bzip2: Deleting output file ABS_MTM_By_Trade.bz2, if it exists.
    bzip2: WARNING: deletion of output file (apparently) failed.

    and of the 29 files in *, this error occurs randomly for only say five
    of the files and the others are ok ??! The files are definitely there.

    Any ideas ?

    Thanks
    Simon
    Simon O, Jun 15, 2006
    #1
    1. Advertising

  2. Simon O wrote:
    >
    > I have this Unix Borne shell script that does something like this in a
    > loop :
    >
    > for FILE in *
    > do
    > ROW_COUNT_HEADER=`...`
    > if [ ${ROW_COUNT_HEADER} -ne 0 ]
    > then
    > ...
    > # 1) OLD Option with Sed :
    > # cat ${FILE} | cut -f1-32 | sed "s/$/ ${FILE}/g" >> file.dat
    > # 2) NEW Option with Perl :
    > cat ${FILE} | cut -f1-32 | perl -ple "s/$/\t${FILE}/g" >>
    > file.dat


    You don't need the /g option because /$/ only occurs once in every line.

    Or you could do that like:

    perl -F'\t' -lane'splice @F, 32; print join "\t", @F, $ARGV' ${FILE}
    >> file.dat



    > fi
    > bzip2 ${FILE}
    > mv ${FILE}.bz2 ${LATEST_ARCHIVE}/IGCPDATA/
    > done
    >
    > I was having problems with using sed, so changed to option 2) using
    > perl which then gave a really weird error. It does what it's supposed
    > to but when it gets to the bzip / move section, it gives an error as
    > though it's still busy with ${FILE} ?? - being driving me MAD ! Almost
    > as though Perl is has said it's finished but is not ?
    >
    > The error is this :
    >
    > bzip2: I/O or other error, bailing out. Possible reason follows.
    > bzip2: No such file or directory
    > Input file = ABS_MTM_By_Trade, output file =
    > ABS_MTM_By_Trade.bz2
    > bzip2: Deleting output file ABS_MTM_By_Trade.bz2, if it exists.
    > bzip2: WARNING: deletion of output file (apparently) failed.


    I don't know enough about bzip to be able to diagnose its errors.



    John
    --
    use Perl;
    program
    fulfillment
    John W. Krahn, Jun 15, 2006
    #2
    1. Advertising

  3. Simon O

    Simon O Guest

    Thanks John for the alternatives.

    Not sure that would solve the problem as the error seems to be in the
    way the shell passes data to Perl and get's it back.

    Bzip - could be any other command - gzip or mv, for that matter - seems
    as though the file is locked by Perl ?

    I thought that because it's unix | perl | unix I thought the file
    handling would all be done by Unix ?

    Simon

    John W. Krahn wrote:
    > Simon O wrote:
    > >
    > > I have this Unix Borne shell script that does something like this in a
    > > loop :
    > >
    > > for FILE in *
    > > do
    > > ROW_COUNT_HEADER=`...`
    > > if [ ${ROW_COUNT_HEADER} -ne 0 ]
    > > then
    > > ...
    > > # 1) OLD Option with Sed :
    > > # cat ${FILE} | cut -f1-32 | sed "s/$/ ${FILE}/g" >> file.dat
    > > # 2) NEW Option with Perl :
    > > cat ${FILE} | cut -f1-32 | perl -ple "s/$/\t${FILE}/g" >>
    > > file.dat

    >
    > You don't need the /g option because /$/ only occurs once in every line.
    >
    > Or you could do that like:
    >
    > perl -F'\t' -lane'splice @F, 32; print join "\t", @F, $ARGV' ${FILE}
    > >> file.dat

    >
    >
    > > fi
    > > bzip2 ${FILE}
    > > mv ${FILE}.bz2 ${LATEST_ARCHIVE}/IGCPDATA/
    > > done
    > >
    > > I was having problems with using sed, so changed to option 2) using
    > > perl which then gave a really weird error. It does what it's supposed
    > > to but when it gets to the bzip / move section, it gives an error as
    > > though it's still busy with ${FILE} ?? - being driving me MAD ! Almost
    > > as though Perl is has said it's finished but is not ?
    > >
    > > The error is this :
    > >
    > > bzip2: I/O or other error, bailing out. Possible reason follows.
    > > bzip2: No such file or directory
    > > Input file = ABS_MTM_By_Trade, output file =
    > > ABS_MTM_By_Trade.bz2
    > > bzip2: Deleting output file ABS_MTM_By_Trade.bz2, if it exists.
    > > bzip2: WARNING: deletion of output file (apparently) failed.

    >
    > I don't know enough about bzip to be able to diagnose its errors.
    >
    >
    >
    > John
    > --
    > use Perl;
    > program
    > fulfillment
    Simon O, Jun 15, 2006
    #3
  4. Simon O

    Dr.Ruud Guest

    Simon O schreef:


    > for FILE in *
    > do
    > cat ${FILE} | cut -f1-32 | perl -ple "s/$/\t${FILE}/g"
    > >> file.dat

    > done


    Probably unrelated to your problem, but the /g modifier is not right.
    Same with the sed command. In both cases, the substitution is done per
    line.

    You don't need cut:

    for FILE in *
    do
    perl -ple "s/^([^\t]*(?:\t[^\t]*){0,31}).*$/\$1\t$FILE/" \
    "$FILE" >> file.dat
    done

    You can of course also do that shell-for-loop in Perl (the language).
    You can even get rid of the shell-script and do it all with perl (the
    implementation), which would give you better control over what goes
    wrong. :)

    --
    Affijn, Ruud

    "Gewoon is een tijger."
    Dr.Ruud, Jun 15, 2006
    #4
  5. Simon O

    Mumia W. Guest

    Simon O wrote:
    > Hi there
    >
    > I have this Unix Borne shell script that does something like this in a
    > loop :
    >
    > for FILE in *
    > do
    > ROW_COUNT_HEADER=`...`
    > if [ ${ROW_COUNT_HEADER} -ne 0 ]
    > then
    > ...
    > # 1) OLD Option with Sed :
    > # cat ${FILE} | cut -f1-32 | sed "s/$/ ${FILE}/g" >> file.dat
    > # 2) NEW Option with Perl :
    > cat ${FILE} | cut -f1-32 | perl -ple "s/$/\t${FILE}/g" >>
    > file.dat


    Perhaps you need to backslash escape that first '$':
    s/\$/\t${FILE}/g

    Or you could just print:
    perl -nle "print; print q[ ${FILE}]"


    > [...]
    Mumia W., Jun 15, 2006
    #5
  6. Simon O wrote:
    > Thanks John for the alternatives.
    >
    > Not sure that would solve the problem as the error seems to be in the
    > way the shell passes data to Perl and get's it back.
    >
    > Bzip - could be any other command - gzip or mv, for that matter - seems
    > as though the file is locked by Perl ?


    No, Perl does not lock the file, and even if you had explicitly flock()ed the
    file, locking files in Unix is only advisory. Besides which, in your example,
    there is no physical file to lock because the data is being piped to perl from
    another program.


    > I thought that because it's unix | perl | unix I thought the file
    > handling would all be done by Unix ?


    Each process in the pipe, be it cut or sed or awk or tr or perl, etc., treats
    the data the same way.


    John
    --
    use Perl;
    program
    fulfillment
    John W. Krahn, Jun 15, 2006
    #6
  7. Simon O wrote:
    > Hi there
    >
    > I have this Unix Borne shell script that does something like this in a
    > loop :
    >
    > for FILE in *
    > do
    > ROW_COUNT_HEADER=`...`
    > if [ ${ROW_COUNT_HEADER} -ne 0 ]
    > then
    > ...
    > # 1) OLD Option with Sed :
    > # cat ${FILE} | cut -f1-32 | sed "s/$/ ${FILE}/g" >> file.dat
    > # 2) NEW Option with Perl :
    > cat ${FILE} | cut -f1-32 | perl -ple "s/$/\t${FILE}/g" >>
    > file.dat
    > fi
    > bzip2 ${FILE}
    > mv ${FILE}.bz2 ${LATEST_ARCHIVE}/IGCPDATA/
    > done
    >
    > I was having problems with using sed, so changed to option 2) using
    > perl which then gave a really weird error. It does what it's supposed
    > to but when it gets to the bzip / move section, it gives an error as
    > though it's still busy with ${FILE} ?? - being driving me MAD ! Almost
    > as though Perl is has said it's finished but is not ?
    >
    > The error is this :
    >
    > bzip2: I/O or other error, bailing out. Possible reason follows.
    > bzip2: No such file or directory
    > Input file = ABS_MTM_By_Trade, output file =
    > ABS_MTM_By_Trade.bz2
    > bzip2: Deleting output file ABS_MTM_By_Trade.bz2, if it exists.
    > bzip2: WARNING: deletion of output file (apparently) failed.
    >
    > and of the 29 files in *, this error occurs randomly for only say five
    > of the files and the others are ok ??! The files are definitely there.
    >


    Maybe you could wrap a sleep/retry loop around `bzip2 ${FILE}` in case
    this is just a transient, recoverable error. This is unrelated to Perl
    though so you might want to try one of the shell groups for followups.

    hth,
    --
    Charles DeRykus
    Charles DeRykus, Jun 15, 2006
    #7
  8. Simon O

    Simon O Guest

    Gentlemen,

    Thanks for all your helpful suggestions, I've justed joined this group
    and am well impressed. I think I've now found a solution, though I
    still don't understand why there was a problem.

    To reduce rowcounts, for testing purposes, I had put a head in the
    chain of piped commands as follows :

    cat ${FILE} | cut -f1-32 | head | perl -ple "s/$/\t${FILE}/" >>
    file.dat

    When I took it out, it worked fine !?? Confused ..

    Sure, my Perl is not great and as always with Perl there are loads of
    ways of killing the cat, but it works !.

    Thanks
    Simon

    John W. Krahn wrote:
    > Simon O wrote:
    > > Thanks John for the alternatives.
    > >
    > > Not sure that would solve the problem as the error seems to be in the
    > > way the shell passes data to Perl and get's it back.
    > >
    > > Bzip - could be any other command - gzip or mv, for that matter - seems
    > > as though the file is locked by Perl ?

    >
    > No, Perl does not lock the file, and even if you had explicitly flock()ed the
    > file, locking files in Unix is only advisory. Besides which, in your example,
    > there is no physical file to lock because the data is being piped to perl from
    > another program.
    >
    >
    > > I thought that because it's unix | perl | unix I thought the file
    > > handling would all be done by Unix ?

    >
    > Each process in the pipe, be it cut or sed or awk or tr or perl, etc., treats
    > the data the same way.
    >
    >
    > John
    > --
    > use Perl;
    > program
    > fulfillment
    Simon O, Jun 15, 2006
    #8
  9. Simon O

    Mumia W. Guest

    Mumia W. wrote:
    > Simon O wrote:
    >> [...]
    >> # 1) OLD Option with Sed :
    >> # cat ${FILE} | cut -f1-32 | sed "s/$/ ${FILE}/g" >> file.dat
    >> # 2) NEW Option with Perl :
    >> cat ${FILE} | cut -f1-32 | perl -ple "s/$/\t${FILE}/g" >>
    >> file.dat

    >
    > Perhaps you need to backslash escape that first '$':
    > s/\$/\t${FILE}/g
    >


    And don't forget that your ${FILE} might contain slashes that conflict
    with the s/// operator's slashes, so use another delimiter:

    s,\$,\t${FILE},g

    You can do something like this with sed also.
    Mumia W., Jun 15, 2006
    #9
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Replies:
    2
    Views:
    5,660
    Jonathan Bromley
    Feb 18, 2005
  2. dpackwood
    Replies:
    3
    Views:
    1,767
  3. Larry
    Replies:
    1
    Views:
    94
    Martien Verbruggen
    Feb 3, 2005
  4. moongeegee

    execute a shell script in a shell script

    moongeegee, Dec 3, 2007, in forum: Perl Misc
    Replies:
    2
    Views:
    239
    Ben Morrow
    Dec 4, 2007
  5. boman
    Replies:
    13
    Views:
    256
    boman
    Aug 27, 2009
Loading...

Share This Page