source filters at runtime

[Xavier Noria]

Looks like source filters do not filter at runtime:

In this example the second pass is not seen by perl:

% perl -wle 'pass;
use Acme:ythonic;
pass
'
Unquoted string "pass" may clash with future reserved word at -e line
1.
Useless use of a constant in void context at -e line 1.

but in this one it is not filtered out:

% perl -wle 'pass;
quote> require Acme:ythonic;
quote> pass
quote> '
Unquoted string "pass" may clash with future reserved word at -e line
1.
Unquoted string "pass" may clash with future reserved word at -e line
4.
Useless use of a constant in void context at -e line 1.
Useless use of a constant in void context at -e line 3.

Since perlfilter says "require" opens a source stream as well as "use"
and also "A source filter is a special kind of Perl module that
intercepts and modifies a source stream before it reaches the parser."
I guessed source filters would filter if they were required (not that
perlfilter says it). But the experiments prove me wrong.

I imagine that since the compilation was already done nothing remains
to be parsed, but I don't know perl internals so I would like to
confirm the reason that's that way.
Why source filters don't filter with require or eval STRING?

-- fxn

 

[Xavier Noria]

I realize that trial with "pass" could not be relevant because the
warning could be being issued at compile time. Anyway, this might a
simpler example of a filter that does not work if required:

% cat Foo.pm
package Foo;
use Filter::Simple sub { s/foo/bar/ };
1;
% perl -wle '
quote> use Foo;
quote> print "foo"
quote> '
bar
% perl -wle '
require Foo;
print "foo"
'
foo

-- fxn

 

[Tassilo v. Parseval]

Also sprach Xavier Noria:

Looks like source filters do not filter at runtime:

Of course not. As I've written elsewhere, they are inserted at a very
early stage, before the parser and lexer kicks in.

In this example the second pass is not seen by perl:

% perl -wle 'pass;
use Acme:ythonic;
pass
'
Unquoted string "pass" may clash with future reserved word at -e line
1.
Useless use of a constant in void context at -e line 1.

but in this one it is not filtered out:

% perl -wle 'pass;
quote> require Acme:ythonic;
quote> pass
quote> '
Unquoted string "pass" may clash with future reserved word at -e line
1.
Unquoted string "pass" may clash with future reserved word at -e line
4.
Useless use of a constant in void context at -e line 1.
Useless use of a constant in void context at -e line 3.

Since perlfilter says "require" opens a source stream as well as "use"
and also "A source filter is a special kind of Perl module that
intercepts and modifies a source stream before it reaches the parser."
I guessed source filters would filter if they were required (not that
perlfilter says it). But the experiments prove me wrong.

They can only filter when they happen at compiletime (preferably early
at compiletime). 'require' happens at runtime, 'use' at compiletime. And
there you have the reason why the source filter in the above has no
effect. If you insist on 'require', you can do:

BEGIN { require Acme:ythonic }

That, however, has no real advantage over 'use Acme:ythonic'.

I imagine that since the compilation was already done nothing remains
to be parsed, but I don't know perl internals so I would like to
confirm the reason that's that way.
Why source filters don't filter with require or eval STRING?

When you include a source filter in your script, on the inside perl
pushes the filter onto a stack of filters. You can have more than one
source filter. When you do that early enough (i.e. at compiletime), you
can have your script filtered through several filters:

ethan@ethan:~$ cat Filter[12].pm
package Filter1;
use Filter::Simple;

FILTER { s/foo/bar/g }

1;
package Filter2;
use Filter::Simple;

FILTER { s/bar/foo/g }

1;
ethan@ethan:~$ perl -MFilter1 -e 'print "foo\n"'
bar
ethan@ethan:~$ perl -MFilter1 -MFilter2 -e 'print "foo\n"'
foo

As you see, Filter2 reverses the effect of Filter1. So when you include
Filter1 before Filter2, the two filters are applied in that order.

In the source of the perl parse, there are hooks in yylex (the functions
that returns the next token) for any installed filter. The hook is a
call to S_filter_gets which always returns the next line of the stream
to be tokenized. If a source filter is installed, S_filter_gets will
delegate the work of retrieving the next line to Perl_filter_read which
then gets the next line and applies all the available source filters to
that line before returning it.

It should be self-understood that it is impossible to apply a source
filter after the source has already been tokenized and parsed. And that
is the reason why it has to happen at compiletime. Therefore, it only
works with 'use' or 'require' put into a BEGIN block.

Think of a source filter as the equivalent to the C pre-processor. It is
isn't quite the same as the cpp processes the whole file before feeding
it to the compiler whereas a source filter is integrated into the
process of tokenizing the stream. However, they both have to happen
'very early'.

A final note: Source filters were not originally added to perl in order
to make Acme:: modules feasible. That was just a side-effect. They were
added so that perl can also parse files in an odd format, such as being
in UTF-16 or having Windows newlines.

Tassilo

 

[Xavier Noria]

Thank you very much! I knew how filters work in general (I wrote
Acme:ythonic in fact), but lacked the C side of this.

It should be self-understood that it is impossible to apply
a source filter after the source has already been tokenized
and parsed. And that is the reason why it has to happen at
compiletime. Therefore, it only works with 'use' or 'require'
put into a BEGIN block.

The problem to self-understand that (that originated I wanted to
doubled check my assumption) is that from the view of a person that
does not know perl's internals, as me, there's no reason to assume perl
cannot remember the necessary context to be able to reenter the
compilation process and redo the entire optree at runtime. In fact I
don't know whether that's technically nonsense, or doable but really
hard, or easily doable but nobody wrote it, ....

Not that my question meant I think that would be desirable, I was
preparing a talk[*] about source filters and happened that l couldn't
explain technically why they didn't work at runtime for certain.

Thank you Tassilo!

-- fxn
[*] In Catalan: http://www.hashref.com/tlk/fdc/filtres_de_codi.pdf