filecmp.dircmp performance

Discussion in 'Python' started by dads, Jan 8, 2011.

  1. dads

    dads Guest

    I'm creating a one way sync program, it's to automate backing up data
    over the wan from our shops to a server at head office. It uses
    filecmp.dircmp() but the performance seems poor to me.

    for x in dc.diff_files:
    srcfp = os.path.join(src, x)
    self.fn777(srcfp)
    if os.path.isfile(srcfp):
    try:
    shutil.copy2(srcfp, dst)
    self.lg.add_diffiles(src, x)
    except Exception, e:
    self.lg.add_errors(e)

    I tested it at a store which is only around 50 miles away on a 10Mbps
    line, the directory has 59 files that are under 100KB. When it gets to
    dc.diff_files it takes 15mins to complete. Looking at the filecmp.py
    it's only using os.stat, it seems excessively long.

    code:
    http://pastebin.com/QskXGDQT
    dads, Jan 8, 2011
    #1
    1. Advertising

  2. dads

    Peter Otten Guest

    dads wrote:

    > I'm creating a one way sync program, it's to automate backing up data
    > over the wan from our shops to a server at head office. It uses
    > filecmp.dircmp() but the performance seems poor to me.
    >
    > for x in dc.diff_files:
    > srcfp = os.path.join(src, x)
    > self.fn777(srcfp)
    > if os.path.isfile(srcfp):
    > try:
    > shutil.copy2(srcfp, dst)
    > self.lg.add_diffiles(src, x)
    > except Exception, e:
    > self.lg.add_errors(e)
    >
    > I tested it at a store which is only around 50 miles away on a 10Mbps
    > line, the directory has 59 files that are under 100KB. When it gets to
    > dc.diff_files it takes 15mins to complete. Looking at the filecmp.py
    > it's only using os.stat, it seems excessively long.


    As a baseline it would be interesting to see how long it takes to copy those
    59 files using system tools.

    However, there are efficient tools out there that work hard to reduce the
    traffic over the net which is likely to be the bottleneck. I suggest that
    you have have a look at

    http://en.wikipedia.org/wiki/Rsync
    Peter Otten, Jan 8, 2011
    #2
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Robin Siebler

    fimecmp.dircmp

    Robin Siebler, Sep 14, 2004, in forum: Python
    Replies:
    0
    Views:
    320
    Robin Siebler
    Sep 14, 2004
  2. Ivan Van Laningham

    Slight discrepancy with filecmp.cmp

    Ivan Van Laningham, Apr 18, 2005, in forum: Python
    Replies:
    3
    Views:
    415
    Dan Sommers
    Apr 18, 2005
  3. =?iso-8859-1?B?TWF0dGlhcyBCcuRuZHN0cvZt?=

    filecmp.cmp() cache

    =?iso-8859-1?B?TWF0dGlhcyBCcuRuZHN0cvZt?=, Feb 15, 2007, in forum: Python
    Replies:
    5
    Views:
    395
    Steve Holden
    Feb 16, 2007
  4. Replies:
    1
    Views:
    831
    Carl Banks
    Sep 7, 2010
  5. Replies:
    2
    Views:
    228
    Paul Lalli
    Jan 4, 2006
Loading...

Share This Page