Presumably an import is no faster or slower than opening a file?

[tinnews]

I'm trying to minimise the overheads of a small Python utility, I'm
not really too fussed about how fast it is but I would like to
minimise its loading effect on the system as it could be called lots
of times (and, no, I don't think there's an easy way of keeping it
running and using the same copy repeatedly).

It needs a configuration file of some sort which I want to keep
separate from the code, is there thus anything to choose between a
configuration file that I read after:-

f = open("configFile", 'r')

.... and importing a configuration written as python dictionaries or
whatever:-

import myConfig

 

[bruno.desthuilliers]

I'm trying to minimise the overheads of a small Python utility, I'm
not really too fussed about how fast it is but I would like to
minimise its loading effect on the system as it could be called lots
of times (and, no, I don't think there's an easy way of keeping it
running and using the same copy repeatedly).

It needs a configuration file of some sort which I want to keep
separate from the code, is there thus anything to choose between a
configuration file that I read after:-

f = open("configFile", 'r')

... and importing a configuration written as python dictionaries or
whatever:-

import myConfig

In both cases, you'll have to open a file. In the first case, you'll
have to parse it each time the script is executed. In the second case,
the first import will take care of compiling the python source to byte-
code and save it in a myConfig.pyc file. As long as the myConfig.py
does not change, subsequent import will directly use the .pyc, so
you'll save on the parsing/compiling time. FWIW, you can even
"manually" compile the myConfig.py file, after each modification, and
only keep the myConfig.pyc in your python path, so you'll never get
the small overhead of the "search .py / search .pyc / compare
modification time / compile / save" cycle.

While we're at it, if you worry about the "overhead" of loading the
conf file, you'd better make sure that either you force the script
compilation and keep the .py out of the path, or at least keep the
script.py file as lightweight as possible (ie : "import somelib;
somelib.main()", where all the real code is in somelib.py), since by
default, only imported modules get their bytecode saved to .pyc file.

Now I may be wrong but I seriously doubt it'll make a huge difference
anyway, and if so, you'd really preferer to have a long running
process to avoid the real overhead of launching a Python interpreter.

My 2 cents...

 

[tinnews]

In both cases, you'll have to open a file. In the first case, you'll
have to parse it each time the script is executed. In the second case,
the first import will take care of compiling the python source to byte-
code and save it in a myConfig.pyc file. As long as the myConfig.py
does not change, subsequent import will directly use the .pyc, so
you'll save on the parsing/compiling time. FWIW, you can even
"manually" compile the myConfig.py file, after each modification, and
only keep the myConfig.pyc in your python path, so you'll never get
the small overhead of the "search .py / search .pyc / compare
modification time / compile / save" cycle.

While we're at it, if you worry about the "overhead" of loading the
conf file, you'd better make sure that either you force the script
compilation and keep the .py out of the path, or at least keep the
script.py file as lightweight as possible (ie : "import somelib;
somelib.main()", where all the real code is in somelib.py), since by
default, only imported modules get their bytecode saved to .pyc file.

Now I may be wrong but I seriously doubt it'll make a huge difference
anyway, and if so, you'd really preferer to have a long running
process to avoid the real overhead of launching a Python interpreter.

Thanks for the comments, mostly about what I was thinking too but I
just wanted to check that I'm not missing anything really obvious.

 

[Paul McGuire]

I'm trying to minimise the overheads of a small Python utility, I'm
not really too fussed about how fast it is but I would like to
minimise its loading effect on the system as it could be called lots
of times (and, no, I don't think there's an easy way of keeping it
running and using the same copy repeatedly).

It needs a configuration file of some sort which I want to keep
separate from the code, is there thus anything to choose between a
configuration file that I read after:-

    f = open("configFile", 'r')

... and importing a configuration written as python dictionaries or
whatever:-

    import myConfig

Chris -

The question is less an issue of the file overhead (both must open a
file, read its contents, and then close it) than what is done with the
file contents afterwards.

A config file containing .INI-style or whatever content will need to
be parsed into Python data/objects, and likely use Python code to do
so. An import will use the Python compiler itself, using optimized
compiled C code to do the parsing and data/object construction. But I
think you would only see the distinction in a config file of
substantial size or complexity. If you think this will make a
substantial difference in performance, then code up a test case and
time it.

In general, I'd say that splitting performance hairs to tip a design
choice one way or another is a misguided premature optimization.

-- Paul

 

[tinnews]

Chris -

The question is less an issue of the file overhead (both must open a
file, read its contents, and then close it) than what is done with the
file contents afterwards.

A config file containing .INI-style or whatever content will need to
be parsed into Python data/objects, and likely use Python code to do
so. An import will use the Python compiler itself, using optimized
compiled C code to do the parsing and data/object construction. But I
think you would only see the distinction in a config file of
substantial size or complexity. If you think this will make a
substantial difference in performance, then code up a test case and
time it.

In general, I'd say that splitting performance hairs to tip a design
choice one way or another is a misguided premature optimization.

I quite agree (about the splitting hairs bit that is), as I said
before I just wanted to check that I wasn't missing anything really
obvious and, thus, that there probably isn't much to choose between
the two approaches. Therefore I'll decide which way to do it on the
basis of 'usability'.