Python Makefiles... are they possible?

Discussion in 'Python' started by Malcolm White, Feb 13, 2013.

  1. I have written a piece of code that will be part of a larger repository of related programs. Within this repository, it is standard to issue a 'make' command to compile any desired program. Is it possible to create a Makefileto compile a simple Python program? Based on what I have come across so far, this is not (at least not typically) the way things are done with Python..

    Thanks for any pointers in the right direction!
    Malcolm White, Feb 13, 2013
    #1
    1. Advertising

  2. On Tue, Feb 12, 2013 at 7:44 PM, Malcolm White <> wrote:

    > I have written a piece of code that will be part of a larger repository of
    > related programs. Within this repository, it is standard to issue a 'make'
    > command to compile any desired program. Is it possible to create a Makefile
    > to compile a simple Python program? Based on what I have come across so
    > far, this is not (at least not typically) the way things are done with
    > Python.
    >
    > Thanks for any pointers in the right direction!
    >


    Python is an interpreted language. The source code is 'compiled' into
    bytecode when invoked. Make is a process that converts source code to
    compiled code, so it doesn't apply to python

    > --
    > http://mail.python.org/mailman/listinfo/python-list
    >




    --
    Joel Goldstick
    http://joelgoldstick.com
    Joel Goldstick, Feb 13, 2013
    #2
    1. Advertising

  3. On 13 February 2013 00:44, Malcolm White <> wrote:
    > I have written a piece of code that will be part of a larger repository of related programs. Within this repository, it is standard to issue a 'make' command to compile any desired program. Is it possible to create a Makefile to compile a simple Python program? Based on what I have come across so far, this is not (at least not typically) the way things are done with Python.


    You can use a Makefile for anything you want in a Python project.
    However Python code is not (typically) compiled so it is not common
    practise to compile it with or without a Makefile. When part of a
    Python project is compiled because, for example it bundles some C code
    to be used within Python, the compilation needs to performed in way
    that will be compatible with Python so the process is normally
    controlled by Python, through a setup.py file. In this case
    compilation is done with something like 'python setup.py build' (Of
    course there's nothing to stop you from adding that command to a
    Makefile and invoking it with 'make').

    I often use Makefiles in Python projects for other purposes, though,
    such as running tests with 'make test' or building documentation with
    'make doc'.


    Oscar
    Oscar Benjamin, Feb 13, 2013
    #3
  4. Malcolm White

    Roy Smith Guest

    In article <>,
    Oscar Benjamin <> wrote:

    > On 13 February 2013 00:44, Malcolm White <> wrote:
    > > I have written a piece of code that will be part of a larger repository of
    > > related programs. Within this repository, it is standard to issue a 'make'
    > > command to compile any desired program. Is it possible to create a Makefile
    > > to compile a simple Python program? Based on what I have come across so
    > > far, this is not (at least not typically) the way things are done with
    > > Python.

    >
    > You can use a Makefile for anything you want in a Python project.
    > However Python code is not (typically) compiled so it is not common
    > practise to compile it with or without a Makefile. When part of a
    > Python project is compiled because, for example it bundles some C code
    > to be used within Python, the compilation needs to performed in way
    > that will be compatible with Python so the process is normally
    > controlled by Python, through a setup.py file. In this case
    > compilation is done with something like 'python setup.py build' (Of
    > course there's nothing to stop you from adding that command to a
    > Makefile and invoking it with 'make').
    >
    > I often use Makefiles in Python projects for other purposes, though,
    > such as running tests with 'make test' or building documentation with
    > 'make doc'.


    One thing we do in our Makefiles is "find . -name '*.pyc' | xargs rm".
    It avoids all sorts of nasty and hard to track down bugs (consider what
    happens if you move a .py file from one place in your source tree to
    another and leave the old .pyc behind).
    Roy Smith, Feb 13, 2013
    #4
  5. On Tue, 12 Feb 2013 20:06:35 -0500, Roy Smith wrote:

    > One thing we do in our Makefiles is "find . -name '*.pyc' | xargs rm".
    > It avoids all sorts of nasty and hard to track down bugs (consider what
    > happens if you move a .py file from one place in your source tree to
    > another and leave the old .pyc behind).



    How often do you move files around in the source tree? Meanwhile, *every*
    time you run make, you take a performance hit on every Python module in
    your project, whether it has moved or not.

    Seems to me like a fairly heavy-handed response for something quite rare,
    but I suppose that depends on how often you run make.




    --
    Steven
    Steven D'Aprano, Feb 13, 2013
    #5
  6. >> One thing we do in our Makefiles is "find . -name '*.pyc' | xargs rm".
    >> It avoids all sorts of nasty and hard to track down bugs (consider what
    >> happens if you move a .py file from one place in your source tree to
    >> another and leave the old .pyc behind).

    >
    > How often do you move files around in the source tree? Meanwhile, *every*
    > time you run make, you take a performance hit on every Python module in
    > your project, whether it has moved or not.
    >
    > Seems to me like a fairly heavy-handed response for something quite rare,
    > but I suppose that depends on how often you run make.


    If the performance hit doesn't really matter.

    Then simply walk the build tree, compare time date stamps, anything that doesn't match up in the make directory, gets deleted. Anything that has different Date Created / Date Modified time from the build tree match, get's deleted.

    This way, we are preserving any files that should be identical. But there should be some mechanism documented to forcibly clear the build cache.

    - Benjamin
    Benjamin Schollnick, Feb 13, 2013
    #6
  7. Malcolm White

    Dave Angel Guest

    On 02/13/2013 12:54 AM, Steven D'Aprano wrote:
    > On Tue, 12 Feb 2013 20:06:35 -0500, Roy Smith wrote:
    >
    >> One thing we do in our Makefiles is "find . -name '*.pyc' | xargs rm".
    >> It avoids all sorts of nasty and hard to track down bugs (consider what
    >> happens if you move a .py file from one place in your source tree to
    >> another and leave the old .pyc behind).

    >
    >
    > How often do you move files around in the source tree? Meanwhile, *every*
    > time you run make, you take a performance hit on every Python module in
    > your project, whether it has moved or not.
    >
    > Seems to me like a fairly heavy-handed response for something quite rare,
    > but I suppose that depends on how often you run make.
    >
    >
    >
    >


    That's why I assumed that his command was triggered by the "make clean"
    command.


    --
    DaveA
    Dave Angel, Feb 13, 2013
    #7
  8. Malcolm White

    Roy Smith Guest

    In article <511b2a7c$0$11096$>,
    Steven D'Aprano <> wrote:

    > On Tue, 12 Feb 2013 20:06:35 -0500, Roy Smith wrote:
    >
    > > One thing we do in our Makefiles is "find . -name '*.pyc' | xargs rm".
    > > It avoids all sorts of nasty and hard to track down bugs (consider what
    > > happens if you move a .py file from one place in your source tree to
    > > another and leave the old .pyc behind).

    >
    >
    > How often do you move files around in the source tree?


    It has happened enough times to make us look for a solution. Which
    means "more than once".

    > Meanwhile, *every* time you run make, you take a performance hit on
    > every Python module in your project, whether it has moved or not.


    The performance hit is minimal. The hours of tearing out your hair
    trying to figure out why bizarre things are happening is not.

    Another solution would be if there was a flag you could give to Python
    to tell it, "Only import a .pyc if the corresponding .py file exists".
    It's already checking to see if the .py is newer, so this wouldn't even
    cost anything.
    Roy Smith, Feb 13, 2013
    #8
  9. Roy Smith wrote:

    > In article <511b2a7c$0$11096$>,
    > Steven D'Aprano <> wrote:
    >
    >> On Tue, 12 Feb 2013 20:06:35 -0500, Roy Smith wrote:
    >>
    >> > One thing we do in our Makefiles is "find . -name '*.pyc' | xargs rm".
    >> > It avoids all sorts of nasty and hard to track down bugs (consider what
    >> > happens if you move a .py file from one place in your source tree to
    >> > another and leave the old .pyc behind).

    >>
    >>
    >> How often do you move files around in the source tree?

    >
    > It has happened enough times to make us look for a solution. Which
    > means "more than once".


    Maybe the solution is education, not a technical fix.

    I've suspicious of technical fixes for developer problems, because in my
    experience that strategy ends in a race to the bottom where you end up with
    coding standards and procedures that assume everyone is a code monkey who
    can barely spell PC. It doesn't protect the monkeys, because there is no
    end to the ways they can screw up, while the competent developers suffer
    under repressive, B&D procedures that hinder more than help.

    YMMV.

    I prefer to keep the .pyc files, and only remove them when necessary, rather
    than to remove them whether it's necessary or not. It's not just because
    I'm an arrogant SOB who expects my team of developers to know at least more
    than me, therefore if I know enough to look for orphaned .pyc files so
    should they. It's because I am a big believer that your development system
    should be as close as possible to the eventual deployment system as is
    practical. Your app will (probably) use .pyc files when it is deployed, so
    you should do the same when developing. Otherwise you can get bugs in
    deployment that you cannot reproduce in development because the
    environments are different.


    >> Meanwhile, *every* time you run make, you take a performance hit on
    >> every Python module in your project, whether it has moved or not.

    >
    > The performance hit is minimal.


    I guess that depends on the size of your project and how much you care about
    the start up time. But as general advice, no, it may not be minimal.

    [...]


    > Another solution would be if there was a flag you could give to Python
    > to tell it, "Only import a .pyc if the corresponding .py file exists".
    > It's already checking to see if the .py is newer, so this wouldn't even
    > cost anything.


    That's called "Python 3.3" :)



    --
    Steven
    Steven D'Aprano, Feb 14, 2013
    #9
  10. Malcolm White

    Roy Smith Guest

    In article <511c501d$0$6512$c3e8da3$>,
    Steven D'Aprano <> wrote:

    > I prefer to keep the .pyc files, and only remove them when necessary, rather
    > than to remove them whether it's necessary or not. It's not just because
    > I'm an arrogant SOB who expects my team of developers to know at least more
    > than me, therefore if I know enough to look for orphaned .pyc files so
    > should they. It's because I am a big believer that your development system
    > should be as close as possible to the eventual deployment system as is
    > practical. Your app will (probably) use .pyc files when it is deployed, so
    > you should do the same when developing.


    Heh. Our deployment system rolls out all the source code from scratch
    on every deploy.

    > Meanwhile, *every* time you run make, you take a performance hit on
    > every Python module in your project, whether it has moved or not.


    Yup.
    Roy Smith, Feb 14, 2013
    #10
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. sasquatch

    Makefiles for JAVA

    sasquatch, Nov 8, 2003, in forum: Java
    Replies:
    5
    Views:
    730
    Dale King
    Nov 12, 2003
  2. Asfand Yar Qazi
    Replies:
    4
    Views:
    375
    Asfand Yar Qazi
    Nov 12, 2004
  3. Martin M.
    Replies:
    4
    Views:
    331
    Simon Brunning
    Dec 15, 2005
  4. Dave Rudolf
    Replies:
    1
    Views:
    290
    Kai-Uwe Bux
    May 17, 2006
  5. David A. Black
    Replies:
    2
    Views:
    210
    Tim Hunter
    Aug 19, 2004
Loading...

Share This Page