Re: No os.copy()? Why not?

Discussion in 'Python' started by D'Arcy Cain, Apr 3, 2012.

  1. D'Arcy Cain

    D'Arcy Cain Guest

    On 03/28/12 16:12, John Ladasky wrote:
    > I'm looking for a Python (2.7) equivalent to the Unix "cp" command.
    > Since the equivalents of "rm" and "mkdir" are in the os module, I
    > figured I look there. I haven't found anything in the documentation.
    > I am also looking through the Python source code in os.py and its
    > child, posixfile.py.


    cp is not a system command, it's a shell command. Why not just use the
    incredibly simple and portable

    >>>open("outfile", "w").write(open("infile").read())


    put it into a method if you find that too much to type:

    def cp(infile, outfile):
    open(outfile, "w").write(open(infile).read())

    --
    D'Arcy J.M. Cain <> | Democracy is three wolves
    http://www.druid.net/darcy/ | and a sheep voting on
    +1 416 425 1212 (DoD#0082) (eNTP) | what's for dinner.
    IM:
    D'Arcy Cain, Apr 3, 2012
    #1
    1. Advertising

  2. On Tue, 03 Apr 2012 15:46:31 -0400, D'Arcy Cain wrote:

    > On 03/28/12 16:12, John Ladasky wrote:
    >> I'm looking for a Python (2.7) equivalent to the Unix "cp" command.
    >> Since the equivalents of "rm" and "mkdir" are in the os module, I
    >> figured I look there. I haven't found anything in the documentation. I
    >> am also looking through the Python source code in os.py and its child,
    >> posixfile.py.

    >
    > cp is not a system command, it's a shell command. Why not just use the
    > incredibly simple and portable
    >
    > >>>open("outfile", "w").write(open("infile").read())

    >
    > put it into a method if you find that too much to type:
    >
    > def cp(infile, outfile):
    > open(outfile, "w").write(open(infile).read())



    Because your cp doesn't copy the FILE, it copies the file's CONTENTS,
    which are not the same thing.

    Consider:

    * permissions
    * access times
    * file ownership
    * other metadata
    * alternate streams and/or resource fork, on platforms that support them
    * sparse files


    By the time you finish supporting the concept of copying the file itself,
    rather than merely its content, you will have something similar to the
    shutil.copy command -- only less tested.



    --
    Steven
    Steven D'Aprano, Apr 4, 2012
    #2
    1. Advertising

  3. Re: No os.copy()? Why not?

    On Wed, Apr 4, 2012 at 3:53 PM, Steven D'Aprano
    <> wrote:
    > On Tue, 03 Apr 2012 15:46:31 -0400, D'Arcy Cain wrote:
    >
    >> def cp(infile, outfile):
    >>    open(outfile, "w").write(open(infile).read())

    >
    > Because your cp doesn't copy the FILE, it copies the file's CONTENTS,
    > which are not the same thing.


    And, as a subtle point: This method can't create the file "at size". I
    don't know how it'll end up allocating space, but certainly there's no
    opportunity to announce to the OS at file open/create time "please
    allocate X bytes for this file". That may be an utterly trivial point,
    or a crucially vital one.

    ChrisA
    Chris Angelico, Apr 4, 2012
    #3
  4. Steven D'Aprano <> writes:

    > On Tue, 03 Apr 2012 15:46:31 -0400, D'Arcy Cain wrote:
    >
    >> On 03/28/12 16:12, John Ladasky wrote:


    >>> I'm looking for a Python (2.7) equivalent to the Unix "cp" command.


    >> >>>open("outfile", "w").write(open("infile").read())


    > Because your cp doesn't copy the FILE, it copies the file's CONTENTS,
    > which are not the same thing.
    > Consider:
    > * permissions
    > * access times
    > * file ownership
    > * other metadata
    > * alternate streams and/or resource fork, on platforms that support them
    > * sparse files
    > By the time you finish supporting the concept of copying the file itself,
    > rather than merely its content, you will have something similar to the
    > shutil.copy command -- only less tested.


    A minor point, but shutil.copy only "copies" contents and permissions
    (no access times, etc.) You probably mean shutil.copy2.

    And sparse files are really hard to reproduce, at least on Unix: on
    Linux even the system's cp doesn't guarantee sparseness of the copy (the
    manual mentions a "crude heuristic").

    But of course shutil.copy is the best solution to mimic a raw cp.

    -- Alain.
    Alain Ketterlin, Apr 4, 2012
    #4
  5. D'Arcy Cain

    Roy Smith Guest

    On Tue, 03 Apr 2012 15:46:31 -0400, D'Arcy Cain wrote:
    > > cp is not a system command, it's a shell command. Why not just use the
    > > incredibly simple and portable
    > >
    > > >>>open("outfile", "w").write(open("infile").read())


    In article <4f7be1e8$0$29999$c3e8da3$>,
    Steven D'Aprano <> wrote:

    > Because your cp doesn't copy the FILE, it copies the file's CONTENTS,
    > which are not the same thing.


    Not to mention that this will read the entire contents of the file into
    memory at once. Probably don't want to do that with 100 GB of data.

    Slightly off-topic, but are there file systems these days which support
    off-line copying? If I have a disk at the other end of a network link,
    it would be nice to tell the disk to copy a file and tell me when it's
    done. As opposed to dragging all that data over the network just so I
    can buffer it in local memory and shove it right back out the network
    port to the same disk. That kind of stuff used to be standard practice
    in the neanderthalic days of IBM mainframes.
    Roy Smith, Apr 4, 2012
    #5
  6. D'Arcy Cain

    Roy Smith Guest

    In article <-strasbg.fr>,
    Alain Ketterlin <-strasbg.fr> wrote:

    > And sparse files are really hard to reproduce, at least on Unix: on
    > Linux even the system's cp doesn't guarantee sparseness of the copy (the
    > manual mentions a "crude heuristic").


    I imagine the heuristic is to look for blocks of all zeros. The problem
    is, unless you know the block size of the file system, you can only
    guess as to how many zeros in a row you need to look for.

    In the old days, dump/restore used to know about sparse files. But
    things like dump/restore really get inside the file system's kimono. In
    today's world of SANs, WANs, and all sorts of virtual file-system-ish
    things, I would expect that's less common.
    Roy Smith, Apr 4, 2012
    #6
  7. Re: No os.copy()? Why not?

    On Wed, Apr 4, 2012 at 10:08 PM, Roy Smith <> wrote:
    > Slightly off-topic, but are there file systems these days which support
    > off-line copying?  If I have a disk at the other end of a network link,
    > it would be nice to tell the disk to copy a file and tell me when it's
    > done.


    Depends on your network protocol. One of the coolest and oldest tricks
    with FTP is initiating a file transfer from one remote host to
    another; I've never done it but it ought to work with localhost (ie
    two sessions to the same host).

    ChrisA
    Chris Angelico, Apr 4, 2012
    #7
  8. D'Arcy Cain

    Steve Howell Guest

    Re: No os.copy()? Why not?

    On Apr 4, 1:37 am, Chris Angelico <> wrote:
    > On Wed, Apr 4, 2012 at 3:53 PM, Steven D'Aprano
    >
    > <> wrote:
    > > On Tue, 03 Apr 2012 15:46:31 -0400, D'Arcy Cain wrote:

    >
    > >> def cp(infile, outfile):
    > >>    open(outfile, "w").write(open(infile).read())

    >
    > > Because your cp doesn't copy the FILE, it copies the file's CONTENTS,
    > > which are not the same thing.

    >
    > And, as a subtle point: This method can't create the file "at size". I
    > don't know how it'll end up allocating space, but certainly there's no
    > opportunity to announce to the OS at file open/create time "please
    > allocate X bytes for this file". That may be an utterly trivial point,
    > or a crucially vital one.
    >
    > ChrisA


    FWIW shutil.py doesn't do anything particularly fancy with respect to
    creating files "at size", unless I'm missing something:

    http://hg.python.org/cpython/file/2.7/Lib/shutil.py

    Only one level away from copyfile, you have copyfileobj, which is a
    read/write loop:

    46 def copyfileobj(fsrc, fdst, length=16*1024):
    47 """copy data from file-like object fsrc to file-like object
    fdst"""
    48 while 1:
    49 buf = fsrc.read(length)
    50 if not buf:
    51 break
    52 fdst.write(buf)

    ....and that gets called by copyfile, which only does a little bit of
    "os"-related stuff:

    66 def copyfile(src, dst):
    67 """Copy data from src to dst"""
    68 if _samefile(src, dst):
    69 raise Error("`%s` and `%s` are the same file" % (src,
    dst))
    70
    71 for fn in [src, dst]:
    72 try:
    73 st = os.stat(fn)
    74 except OSError:
    75 # File most likely does not exist
    76 pass
    77 else:
    78 # XXX What about other special files? (sockets,
    devices...)
    79 if stat.S_ISFIFO(st.st_mode):
    80 raise SpecialFileError("`%s` is a named pipe" %
    fn)
    81
    82 with open(src, 'rb') as fsrc:
    83 with open(dst, 'wb') as fdst:
    84 copyfileobj(fsrc, fdst)

    The "value add" vs. a simple read/write loop depends on whether you
    want OSError suppressed. The _samefile guard is nice to have, but
    probably unnecessary for many apps.

    I'm sure shutil.copyfile() makes perfect sense for most use cases, and
    it's nice that you can see what it does under the covers pretty
    easily, but it's not rocket science.
    Steve Howell, Apr 4, 2012
    #8
  9. D'Arcy Cain

    Nobody Guest

    On Wed, 04 Apr 2012 08:14:18 -0400, Roy Smith wrote:

    >> And sparse files are really hard to reproduce, at least on Unix: on
    >> Linux even the system's cp doesn't guarantee sparseness of the copy (the
    >> manual mentions a "crude heuristic").

    >
    > I imagine the heuristic is to look for blocks of all zeros.


    Yes. Although it's not really accurate to describe it as a "heuristic".

    With --sparse=always, it will try to make the output sparse regardless of
    whether the input was sparse, replacing any all-zeros block with a hole.

    The default of --sparse=auto will only create a sparse file if the input
    itself is sparse, i.e. if the length of the file rounded up to the nearest
    block exceeds its disk usage.

    Regardless of the --sparse= setting and whether the input was sparse, if
    it tries to create a sparse file it will create holes wherever possible
    rather than attempting to preserve the exact pattern of holes in a sparse
    input file.
    Nobody, Apr 4, 2012
    #9
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Alex
    Replies:
    2
    Views:
    1,206
  2. Mr. SweatyFinger

    why why why why why

    Mr. SweatyFinger, Nov 28, 2006, in forum: ASP .Net
    Replies:
    4
    Views:
    863
    Mark Rae
    Dec 21, 2006
  3. Mr. SweatyFinger
    Replies:
    2
    Views:
    1,769
    Smokey Grindel
    Dec 2, 2006
  4. rdc02271
    Replies:
    24
    Views:
    723
    Jacek Dziedzic
    Dec 27, 2005
  5. Replies:
    26
    Views:
    2,096
    Roland Pibinger
    Sep 1, 2006
Loading...

Share This Page