Merge two directories together

Discussion in 'Python' started by Keith Hughitt, Apr 16, 2010.

  1. Suppose you have two file-trees with common sub-directories but
    different files that you want to merge together, e.g.

    /test/
    /test/a/
    /test/a/file1

    /test2/
    /test2/a/
    /test2/a/file2

    You can easily merge the directories in Linux using the "cp" command:

    cp -r test/* test2/

    While Python provides some helpful methods for moving files and
    directories around (shutil.move, shutil.copytree, etc), none of them
    seem to be able to merge two directories.

    I've looked around some on Google for an alternative to calling cp
    directly, but so far no luck.

    Any ideas?
     
    Keith Hughitt, Apr 16, 2010
    #1
    1. Advertising

  2. Keith Hughitt

    Steven Howe Guest

    Think about using the subprocess module. There are calls made just for
    your. Notably (using command pydoc subprrocess.call) :
    ---------------------
    subprocess.call = call(*popenargs, **kwargs)
    Run command with arguments. Wait for command to complete, then
    return the returncode attribute.

    The arguments are the same as for the Popen constructor. Example:

    retcode = call(["ls", "-l"])
    ---------------------


    Steven Howe


    On 04/16/2010 06:48 AM, Keith Hughitt wrote:
    > Suppose you have two file-trees with common sub-directories but
    > different files that you want to merge together, e.g.
    >
    > /test/
    > /test/a/
    > /test/a/file1
    >
    > /test2/
    > /test2/a/
    > /test2/a/file2
    >
    > You can easily merge the directories in Linux using the "cp" command:
    >
    > cp -r test/* test2/
    >
    > While Python provides some helpful methods for moving files and
    > directories around (shutil.move, shutil.copytree, etc), none of them
    > seem to be able to merge two directories.
    >
    > I've looked around some on Google for an alternative to calling cp
    > directly, but so far no luck.
    >
    > Any ideas?
    >
     
    Steven Howe, Apr 16, 2010
    #2
    1. Advertising

  3. Keith Hughitt

    Dave W. Guest

    > While Python provides some helpful methods for moving files and
    > directories around (shutil.move, shutil.copytree, etc), none of
    > them seem to be able to merge two directories.

    -snip-
    > Any ideas?


    It's not pretty, but you could hack up the original copytree()
    source so it ignores errors from the makedirs() call. Then it
    should work more-or-less like 'cp -r'. This isn't ideal, because
    'OSError' is thrown, which could mean that the dir already exists
    (okay), or it could be a 'real' error, like the current user doesn't
    have permission to write to the destination. (See 'XXX' in the code
    below.)

    Better would be to match on the error string and only ignore the
    'directory exists' error. Unfortunately, the error messages do
    vary per-platform. Here's what I see under Windows when the
    directory already exists:

    WindowsError: [Error 183] Cannot create a file when that file
    already exists: 'test2'

    and under CentOS:

    OSError: [Errno 17] File exists: 'test2'

    The code below seems to work under both Linux and Windows; but I
    didn't really test it much, so handle with care. :-}

    ----------

    def copytree(src, dst, symlinks=False, ignore=None):
    import os
    from shutil import copy2, copystat, Error

    names = os.listdir(src)
    if ignore is not None:
    ignored_names = ignore(src, names)
    else:
    ignored_names = set()

    try:
    os.makedirs(dst)
    except OSError, exc:
    # XXX - this is pretty ugly
    if "file already exists" in exc[1]: # Windows
    pass
    elif "File exists" in exc[1]: # Linux
    pass
    else:
    raise

    errors = []
    for name in names:
    if name in ignored_names:
    continue
    srcname = os.path.join(src, name)
    dstname = os.path.join(dst, name)
    try:
    if symlinks and os.path.islink(srcname):
    linkto = os.readlink(srcname)
    os.symlink(linkto, dstname)
    elif os.path.isdir(srcname):
    copytree(srcname, dstname, symlinks, ignore)
    else:
    copy2(srcname, dstname)
    # XXX What about devices, sockets etc.?
    except (IOError, os.error), why:
    errors.append((srcname, dstname, str(why)))
    # catch the Error from the recursive copytree so that we can
    # continue with other files
    except Error, err:
    errors.extend(err.args[0])
    try:
    copystat(src, dst)
    except WindowsError:
    # can't copy file access times on Windows
    pass
    except OSError, why:
    errors.extend((src, dst, str(why)))
    if errors:
    raise Error, errors
     
    Dave W., Apr 16, 2010
    #3
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Joel Finkel
    Replies:
    0
    Views:
    508
    Joel Finkel
    Sep 12, 2003
  2. Jeffry van de Vuurst
    Replies:
    2
    Views:
    548
    Jeffry van de Vuurst
    Jul 30, 2003
  3. =?Utf-8?B?TGFzc2UgTmlsc3Nvbg==?=

    Multiple bin-directories with virtual directories?

    =?Utf-8?B?TGFzc2UgTmlsc3Nvbg==?=, Nov 9, 2004, in forum: ASP .Net
    Replies:
    0
    Views:
    839
    =?Utf-8?B?TGFzc2UgTmlsc3Nvbg==?=
    Nov 9, 2004
  4. Gregory Ewing
    Replies:
    0
    Views:
    271
    Gregory Ewing
    May 11, 2011
  5. nntp
    Replies:
    9
    Views:
    228
    Anno Siegel
    Oct 12, 2004
Loading...

Share This Page