source filters at runtime

Discussion in 'Perl Misc' started by Xavier Noria, Nov 25, 2004.

  1. Xavier Noria

    Xavier Noria Guest

    Looks like source filters do not filter at runtime:

    In this example the second pass is not seen by perl:

    % perl -wle 'pass;
    use Acme::pythonic;
    pass
    '
    Unquoted string "pass" may clash with future reserved word at -e line
    1.
    Useless use of a constant in void context at -e line 1.

    but in this one it is not filtered out:

    % perl -wle 'pass;
    quote> require Acme::pythonic;
    quote> pass
    quote> '
    Unquoted string "pass" may clash with future reserved word at -e line
    1.
    Unquoted string "pass" may clash with future reserved word at -e line
    4.
    Useless use of a constant in void context at -e line 1.
    Useless use of a constant in void context at -e line 3.

    Since perlfilter says "require" opens a source stream as well as "use"
    and also "A source filter is a special kind of Perl module that
    intercepts and modifies a source stream before it reaches the parser."
    I guessed source filters would filter if they were required (not that
    perlfilter says it). But the experiments prove me wrong.

    I imagine that since the compilation was already done nothing remains
    to be parsed, but I don't know perl internals so I would like to
    confirm the reason that's that way.
    Why source filters don't filter with require or eval STRING?

    -- fxn
     
    Xavier Noria, Nov 25, 2004
    #1
    1. Advertising

  2. Xavier Noria

    Xavier Noria Guest

    I realize that trial with "pass" could not be relevant because the
    warning could be being issued at compile time. Anyway, this might a
    simpler example of a filter that does not work if required:

    % cat Foo.pm
    package Foo;
    use Filter::Simple sub { s/foo/bar/ };
    1;
    % perl -wle '
    quote> use Foo;
    quote> print "foo"
    quote> '
    bar
    % perl -wle '
    require Foo;
    print "foo"
    '
    foo

    -- fxn
     
    Xavier Noria, Nov 25, 2004
    #2
    1. Advertising

  3. Also sprach Xavier Noria:

    > Looks like source filters do not filter at runtime:


    Of course not. As I've written elsewhere, they are inserted at a very
    early stage, before the parser and lexer kicks in.

    > In this example the second pass is not seen by perl:
    >
    > % perl -wle 'pass;
    > use Acme::pythonic;
    > pass
    > '
    > Unquoted string "pass" may clash with future reserved word at -e line
    > 1.
    > Useless use of a constant in void context at -e line 1.
    >
    > but in this one it is not filtered out:
    >
    > % perl -wle 'pass;
    > quote> require Acme::pythonic;
    > quote> pass
    > quote> '
    > Unquoted string "pass" may clash with future reserved word at -e line
    > 1.
    > Unquoted string "pass" may clash with future reserved word at -e line
    > 4.
    > Useless use of a constant in void context at -e line 1.
    > Useless use of a constant in void context at -e line 3.
    >
    > Since perlfilter says "require" opens a source stream as well as "use"
    > and also "A source filter is a special kind of Perl module that
    > intercepts and modifies a source stream before it reaches the parser."
    > I guessed source filters would filter if they were required (not that
    > perlfilter says it). But the experiments prove me wrong.


    They can only filter when they happen at compiletime (preferably early
    at compiletime). 'require' happens at runtime, 'use' at compiletime. And
    there you have the reason why the source filter in the above has no
    effect. If you insist on 'require', you can do:

    BEGIN { require Acme::pythonic }

    That, however, has no real advantage over 'use Acme::pythonic'.

    > I imagine that since the compilation was already done nothing remains
    > to be parsed, but I don't know perl internals so I would like to
    > confirm the reason that's that way.
    > Why source filters don't filter with require or eval STRING?


    When you include a source filter in your script, on the inside perl
    pushes the filter onto a stack of filters. You can have more than one
    source filter. When you do that early enough (i.e. at compiletime), you
    can have your script filtered through several filters:

    ethan@ethan:~$ cat Filter[12].pm
    package Filter1;
    use Filter::Simple;

    FILTER { s/foo/bar/g }

    1;
    package Filter2;
    use Filter::Simple;

    FILTER { s/bar/foo/g }

    1;
    ethan@ethan:~$ perl -MFilter1 -e 'print "foo\n"'
    bar
    ethan@ethan:~$ perl -MFilter1 -MFilter2 -e 'print "foo\n"'
    foo

    As you see, Filter2 reverses the effect of Filter1. So when you include
    Filter1 before Filter2, the two filters are applied in that order.

    In the source of the perl parse, there are hooks in yylex (the functions
    that returns the next token) for any installed filter. The hook is a
    call to S_filter_gets which always returns the next line of the stream
    to be tokenized. If a source filter is installed, S_filter_gets will
    delegate the work of retrieving the next line to Perl_filter_read which
    then gets the next line and applies all the available source filters to
    that line before returning it.

    It should be self-understood that it is impossible to apply a source
    filter after the source has already been tokenized and parsed. And that
    is the reason why it has to happen at compiletime. Therefore, it only
    works with 'use' or 'require' put into a BEGIN block.

    Think of a source filter as the equivalent to the C pre-processor. It is
    isn't quite the same as the cpp processes the whole file before feeding
    it to the compiler whereas a source filter is integrated into the
    process of tokenizing the stream. However, they both have to happen
    'very early'.

    A final note: Source filters were not originally added to perl in order
    to make Acme:: modules feasible. That was just a side-effect. They were
    added so that perl can also parse files in an odd format, such as being
    in UTF-16 or having Windows newlines.

    Tassilo
    --
    $_=q#",}])!JAPH!qq(tsuJ[{@"tnirp}3..0}_$;//::niam/s~=)]3[))_$-3(rellac(=_$({
    pam{rekcahbus})(rekcah{lrePbus})(lreP{rehtonabus})!JAPH!qq(rehtona{tsuJbus#;
    $_=reverse,s+(?<=sub).+q#q!'"qq.\t$&."'!#+sexisexiixesixeseg;y~\n~~dddd;eval
     
    Tassilo v. Parseval, Nov 26, 2004
    #3
  4. Xavier Noria

    Xavier Noria Guest

    Thank you very much! I knew how filters work in general (I wrote
    Acme::pythonic in fact), but lacked the C side of this.

    >It should be self-understood that it is impossible to apply
    >a source filter after the source has already been tokenized
    >and parsed. And that is the reason why it has to happen at
    >compiletime. Therefore, it only works with 'use' or 'require'
    >put into a BEGIN block.


    The problem to self-understand that (that originated I wanted to
    doubled check my assumption) is that from the view of a person that
    does not know perl's internals, as me, there's no reason to assume perl
    cannot remember the necessary context to be able to reenter the
    compilation process and redo the entire optree at runtime. In fact I
    don't know whether that's technically nonsense, or doable but really
    hard, or easily doable but nobody wrote it, ....

    Not that my question meant I think that would be desirable, I was
    preparing a talk[*] about source filters and happened that l couldn't
    explain technically why they didn't work at runtime for certain.

    Thank you Tassilo!

    -- fxn
    [*] In Catalan: http://www.hashref.com/tlk/fdc/filtres_de_codi.pdf
     
    Xavier Noria, Nov 27, 2004
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Fred Kuijper

    Digital filters

    Fred Kuijper, Jul 13, 2003, in forum: VHDL
    Replies:
    1
    Views:
    2,057
    Michael Gallen
    Jul 17, 2003
  2. Frank van Eijkelenburg

    filters in vhdl

    Frank van Eijkelenburg, Sep 30, 2003, in forum: VHDL
    Replies:
    1
    Views:
    757
  3. viswanath

    regarding filters in vhdl

    viswanath, May 17, 2004, in forum: VHDL
    Replies:
    7
    Views:
    8,355
    kjaraline
    Nov 18, 2009
  4. Dieter Vanderelst

    using the Filters DLL (image filters)

    Dieter Vanderelst, Feb 15, 2006, in forum: Python
    Replies:
    1
    Views:
    437
    Michele Petrazzo
    Feb 15, 2006
  5. Lex Williams

    does ruby have source filters?

    Lex Williams, Aug 21, 2008, in forum: Ruby
    Replies:
    4
    Views:
    194
    Tim Hunter
    Aug 21, 2008
Loading...

Share This Page