How to know that two pyc files contain the same code

Discussion in 'Python' started by Gelonida N, Mar 10, 2012.

  1. Gelonida N

    Gelonida N Guest

    Hi,

    I want to know whether two .pyc files are identical.

    With identical I mean whether they contain the same byte code.

    Unfortunately it seems, that .pyc files contain also something like the
    time stamp of the related source file.

    So though two pyc files contain the same byte code, they will not be
    byte identical.

    One option, that I found is to use
    python -m unpyclib.application -d filename.pyc and check whether the
    results are identical.


    However even this will fail if the files were not compiled under the
    same absolute path name as the source filename is contained twice (at
    least for my trivial example) in the disassemblers output.


    Thanks a lot for any other idea.
    Gelonida N, Mar 10, 2012
    #1
    1. Advertising

  2. On Sat, 10 Mar 2012 15:48:48 +0100, Gelonida N wrote:

    > Hi,
    >
    > I want to know whether two .pyc files are identical.
    >
    > With identical I mean whether they contain the same byte code.


    Define "identical" and "the same".

    If I compile these two files:


    # file ham.py
    x = 23
    def func():
    a = 23
    return a + 19



    # file = spam.py
    def func():
    return 42

    tmp = 19
    x = 4 + tmp
    del tmp


    do you expect spam.pyc and ham.pyc to count as "the same"?


    --
    Steven
    Steven D'Aprano, Mar 10, 2012
    #2
    1. Advertising

  3. On Sun, Mar 11, 2012 at 9:52 AM, Steven D'Aprano
    <> wrote:
    > On Sat, 10 Mar 2012 15:48:48 +0100, Gelonida N wrote:
    > Define "identical" and "the same".
    >
    > If I compile these two files:
    >
    >
    > # file ham.py
    > x = 23
    > def func():
    >    a = 23
    >    return a + 19
    >
    >
    >
    > # file = spam.py
    > def func():
    >    return 42
    >
    > tmp = 19
    > x = 4 + tmp
    > del tmp
    >
    >
    > do you expect spam.pyc and ham.pyc to count as "the same"?


    They do not contain the same code. They may contain code which has the
    same effect, but it is not the same code.

    I don't think Python has the level of aggressive optimization that
    would make these compile to the same bytecode, but if it did, then
    they would _become identical_ per the OP's description - that they
    contain identical bytecode. In fact, I think the OP defined it quite
    clearly.

    ChrisA
    Chris Angelico, Mar 11, 2012
    #3
  4. Gelonida N

    Gelonida N Guest

    Hi Steven,

    On 03/10/2012 11:52 PM, Steven D'Aprano wrote:
    > > On Sat, 10 Mar 2012 15:48:48 +0100, Gelonida N wrote:
    > >
    >> >> Hi,
    >> >>
    >> >> I want to know whether two .pyc files are identical.
    >> >>
    >> >> With identical I mean whether they contain the same byte code.

    > >
    > > Define "identical" and "the same".

    Indeed! Identical is not that simple to define and depends on the context.

    One definition of identical, that would suit me at the moment would be:

    If I have two .pyc files, which were the result of a compilation of
    two identical .py files, then I would like to treat these two .pyc files
    as identical,
    even if they were compiled at different times (absolutely necessary)
    and with a different absolute path (would be nice)

    Above definition of identical byte code would also mean, that any error
    message about errors in a given line number would be identical for both
    ..pyc files

    > >
    > > If I compile these two files:
    > >
    > >
    > > # file ham.py
    > > x = 23
    > > def func():
    > > a = 23
    > > return a + 19
    > >
    > >
    > >
    > > # file = spam.py
    > > def func():
    > > return 42
    > >
    > > tmp = 19
    > > x = 4 + tmp
    > > del tmp
    > >
    > >
    > > do you expect spam.pyc and ham.pyc to count as "the same"?
    > >

    For most pythons I would not expect, that ham.py and spam.py would
    result in the same byte code and would thus not even have the same
    performance,

    I agree, though that an efficient compiler might generate the same byte
    code, though I wonder if an optimizing compiler would/should be allowed
    to optimize away the global variable tmp, as it would be visible (though
    only for a short time) in a multithreading environment.

    If the byte code were different in two .pyc files. then I would
    like to have them treated as different .pyc files.

    If by coincidence, the generated btye code were the same, then I
    wouldn't mind, if they were treated as identical, but I wouldn't insist.

    Up to my knowledge Python (or at least C-python) stores line numbers in
    the .pyc files, so that it can report exact line numbers refering to the
    originating source code in case of an exception or for back traces

    So there is the choice to say, that two pyc files with exactly the same
    byte code would be treated identical if white spaces / line numbers of
    their sources were different or the choice to say, that they are
    different.

    Being conservative I'd treat them as different.

    Ideally I'd like to be able depending on my use case to distinguish
    following cases.
    a) .pyc files with identical byte code
    b) .pyc files with identical byte code AND source code line numbers
    c) same as b) AND identical source file names.
    Gelonida N, Mar 11, 2012
    #4
  5. On Sun, 11 Mar 2012 12:15:11 +1100, Chris Angelico wrote:

    > On Sun, Mar 11, 2012 at 9:52 AM, Steven D'Aprano
    > <> wrote:
    >> On Sat, 10 Mar 2012 15:48:48 +0100, Gelonida N wrote: Define
    >> "identical" and "the same".
    >>
    >> If I compile these two files:
    >>
    >>
    >> # file ham.py
    >> x = 23
    >> def func():
    >>    a = 23
    >>    return a + 19
    >>
    >>
    >>
    >> # file = spam.py
    >> def func():
    >>    return 42
    >>
    >> tmp = 19
    >> x = 4 + tmp
    >> del tmp
    >>
    >>
    >> do you expect spam.pyc and ham.pyc to count as "the same"?

    >
    > They do not contain the same code. They may contain code which has the
    > same effect, but it is not the same code.


    To me, they do: they contain a function "func" which takes no arguments
    and returns 42, and a global "x" initialised to 23. Everything else is an
    implementation detail.

    I'm not being facetious. One should be asking what is the *purpose* of
    this question -- is it to detect when two pyc files contain the same
    *interface*, or to determine if they were generated from identical source
    code files (and if the later, do comments and whitespace matter)?

    What if one merely changed the order of definition? Instead of:

    def foo(): pass
    def bar(): pass

    one had this?

    def bar(): pass
    def foo(): pass

    It depends on why the OP cares if they are "identical". I can imagine use-
    cases where the right solution is to forget ideas about identical code,
    and just checksum the files (ignoring any timestamps).


    --
    Steven
    Steven D'Aprano, Mar 11, 2012
    #5
  6. Gelonida N

    Rick Johnson Guest

    On Mar 11, 2:06 am, Steven D'Aprano <steve
    > wrote:

    > I'm not being facetious. [...]


    Just in case anybody does not know already: the "D'A" in "Steven
    D'Aprano" stands for "Devils Advocate"; his real name is "Steven
    Prano".
    Rick Johnson, Mar 11, 2012
    #6
  7. Gelonida N

    Gelonida N Guest

    On 03/11/2012 08:06 AM, Steven D'Aprano wrote:
    > What if one merely changed the order of definition? Instead of:
    >
    > def foo(): pass
    > def bar(): pass
    >
    > one had this?
    >
    > def bar(): pass
    > def foo(): pass
    >
    > It depends on why the OP cares if they are "identical". I can imagine use-
    > cases where the right solution is to forget ideas about identical code,
    > and just checksum the files (ignoring any timestamps).


    I guess this is what I will do for my use case
    Perform a checksum ignoring the time stamp.

    What I did not know though is where the time stamp was located.
    it seems it's in bytes 4-7 for all C-python versions so far.

    What is regrettable though is, that the absolute path name is part of
    the .pyc file, as I do not care


    Following snippet calculates the hash of a .pyc file by just ignoring
    the time stamp:

    import hashlib

    def md5_for_pyc(fname):
    hasher = hashlib.md5()
    with open(fname, 'rb') as fin:
    version = fin.read(4)
    hasher.update(version)
    _tstamp = fin.read(4)
    bytecode = fin.read()
    hasher.update(bytecode)
    return hasher.hexdigest()
    Gelonida N, Mar 11, 2012
    #7
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. gelonida
    Replies:
    1
    Views:
    748
    Gabriel Genellina
    May 6, 2010
  2. Roger Pack
    Replies:
    3
    Views:
    155
    Roger Pack
    Sep 28, 2010
  3. Gelonida N
    Replies:
    1
    Views:
    230
    Bl0ckeduser
    Mar 10, 2012
  4. Andries

    I know, I know, I don't know

    Andries, Apr 23, 2004, in forum: Perl Misc
    Replies:
    3
    Views:
    227
    Gregory Toomey
    Apr 23, 2004
  5. Jason Carlton
    Replies:
    11
    Views:
    246
    Dr J R Stockton
    Dec 8, 2009
Loading...

Share This Page