Parsing text file with #include and #define directives

P

python

I'm parsing a text file for a proprietary product that has the following
2 directives:

#include <somefile>
#define <name> <value>

Defined constants are referenced via <#name#> syntax.

I'm looking for a single text stream that results from processing a file
containing these directives. Even better would be an iterator(?) type
object that tracked file names and line numbers as it returns individual
lines.

Is there a Python parsing library to handle this type of task or am I
better off writing my own?

The effort to write one from scratch doesn't seem too difficult (minus
recursive file and constant loops), but I wanted to avoid re-inventing
the wheel if this type of component already exists.

Thank you,

Malcolm
 
A

Arnaud Delobelle

I'm parsing a text file for a proprietary product that has the following
2 directives:

#include <somefile>
#define <name> <value>

Defined constants are referenced via <#name#> syntax.

I'm looking for a single text stream that results from processing a file
containing these directives. Even better would be an iterator(?) type
object that tracked file names and line numbers as it returns individual
lines.

Is there a Python parsing library to handle this type of task or am I
better off writing my own?

The effort to write one from scratch doesn't seem too difficult (minus
recursive file and constant loops), but I wanted to avoid re-inventing
the wheel if this type of component already exists.

Thank you,

Malcolm

I think it's straightforward enough to be dealt with simply. Here is
a solution that doesn't handle errors but should work with well-formed
input and handles recursive expansions.

expand(filename) returns an iterator over expanded lines in the file,
inserting lines of included files.

import re

def expand(filename):
defines = {}
def define_repl(matchobj):
return defines[matchobj.group(1)]
define_regexp = re.compile('#(.+?)#')
for line in open(filename):
if line.startswith('#include '):
recfilename = line.strip().split(None, 1)[1]
for recline in expand(recfilename):
yield recline
elif line.startswith('#define '):
_, name, value = line.strip().split(None, 2)
defines[name] = value
else:
yield define_regexp.sub(define_repl, line)

It would be easy to modify it to keep track of line numbers and file
names.

HTH
 
P

python

Arnaud,

Wow!!! That's beautiful. Thank you very much!

Malcolm

<snip>

I think it's straightforward enough to be dealt with simply. Here is
a solution that doesn't handle errors but should work with well-formed
input and handles recursive expansions.

expand(filename) returns an iterator over expanded lines in the file,
inserting lines of included files.

import re

def expand(filename):
defines = {}
def define_repl(matchobj):
return defines[matchobj.group(1)]
define_regexp = re.compile('#(.+?)#')
for line in open(filename):
if line.startswith('#include '):
recfilename = line.strip().split(None, 1)[1]
for recline in expand(recfilename):
yield recline
elif line.startswith('#define '):
_, name, value = line.strip().split(None, 2)
defines[name] = value
else:
yield define_regexp.sub(define_repl, line)

It would be easy to modify it to keep track of line numbers and file
names.

</snip>
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
473,744
Messages
2,569,482
Members
44,900
Latest member
Nell636132

Latest Threads

Top