convert ftp.retrbinary to file object? - Python language lacks expression?

Discussion in 'Python' started by Robert, Feb 3, 2005.

  1. Robert

    Robert Guest

    I just tried to convert a (hugh size) ftp.retrbinary run into a
    pseudo-file object with .read(bytes) method in order to not consume
    500MB on a copy operation.

    First I thought, its easy as usual with python using something like
    'yield' or so.

    Yet I didn't manage to do (without using threads or rewriting
    'retrbinary')? Any ideas?

    #### I tried a pattern like:
    ....
    def open(self,ftppath,mode='rb'):
    class FTPFile: #TODO
    ...
    def iter_retr()
    ...
    def callback(blk):
    how-to-yield-from-here-to-iter_retr blk???
    ftp.retrbinary("RETR %s" % relpath,callback)
    def read(self, bytes=-1):
    ...
    self.buf+=self.iter.next()
    ...
    ....
     
    Robert, Feb 3, 2005
    #1
    1. Advertising

  2. Re: convert ftp.retrbinary to file object? - Python language lacksexpression?

    Robert wrote:
    > I just tried to convert a (hugh size) ftp.retrbinary run into a
    > pseudo-file object with .read(bytes) method in order to not consume
    > 500MB on a copy operation.
    >
    > First I thought, its easy as usual with python using something like
    > 'yield' or so.
    >
    > Yet I didn't manage to do (without using threads or rewriting
    > 'retrbinary')? Any ideas?
    >
    > #### I tried a pattern like:
    > ....
    > def open(self,ftppath,mode='rb'):
    > class FTPFile: #TODO
    > ...
    > def iter_retr()
    > ...
    > def callback(blk):
    > how-to-yield-from-here-to-iter_retr blk???
    > ftp.retrbinary("RETR %s" % relpath,callback)
    > def read(self, bytes=-1):
    > ...
    > self.buf+=self.iter.next()
    > ...
    > ....



    Hmmmm this is nearly there I think...:

    import ftplib

    class TransferAbort(Exception): pass

    class FTPFile:
    def __init__(self, server, filename):
    self.server = server
    self.filename = filename
    self.offset = 0

    def callback(self, data):
    self.offset = self.offset + len(data)
    self.data = data
    ## now quit the RETR command?
    raise TransferAbort("stop right now")

    def read(self, amount):
    self.ftp = ftplib.FTP(self.server)
    self.ftp.login()
    try:
    self.ftp.retrbinary("RETR %s" %self.filename, self.callback,
    blocksize=amount,
    rest=self.offset)
    except TransferAbort:
    return self.data


    f = FTPFile("HOSTNAME", "FILENAME")

    print f.read(24)
    print f.read(24)


    I open the ftp connection inside the read method as it caused an error
    (on the second call to read) when I opened it in __init__ ???

    HTH
    Martin
     
    Martin Franklin, Feb 3, 2005
    #2
    1. Advertising

  3. Re: convert ftp.retrbinary to file object? - Python language lacksexpression?

    Martin Franklin wrote:
    > Robert wrote:
    >
    >> I just tried to convert a (hugh size) ftp.retrbinary run into a
    >> pseudo-file object with .read(bytes) method in order to not consume
    >> 500MB on a copy operation.


    [snip]

    >
    >
    > Hmmmm this is nearly there I think...:


    whoops... spoke too soon..

    >
    > import ftplib
    >
    > class TransferAbort(Exception): pass
    >
    > class FTPFile:
    > def __init__(self, server, filename):
    > self.server = server
    > self.filename = filename
    > self.offset = 0
    >
    > def callback(self, data):
    > self.offset = self.offset + len(data)
    > self.data = data
    > ## now quit the RETR command?
    > raise TransferAbort("stop right now")
    >
    > def read(self, amount):
    > self.ftp = ftplib.FTP(self.server)
    > self.ftp.login()



    I needed to insert a time.sleep(0.1) here as the connections were
    falling over themselves - I guess testing with a blocksize of 24
    is a little silly.


    > try:
    > self.ftp.retrbinary("RETR %s" %self.filename, self.callback,
    > blocksize=amount,
    > rest=self.offset)
    > except TransferAbort:
    > return self.data
    >
    >
    > f = FTPFile("HOSTNAME", "FILENAME")
    >
    > print f.read(24)
    > print f.read(24)
    >


    ## new test...

    f = FTPFile("HOSTNAME", "FILENAME")

    while 1:
    data = f.read(24)
    if not data:
    break
    print data,


    >
    > I open the ftp connection inside the read method as it caused an error
    > (on the second call to read) when I opened it in __init__ ???
    >
    > HTH
    > Martin
    >
    >
     
    Martin Franklin, Feb 3, 2005
    #3
  4. Re: convert ftp.retrbinary to file object? - Python language lacksexpression?

    Martin Franklin wrote:
    > Martin Franklin wrote:
    >
    >> Robert wrote:
    >>
    >>> I just tried to convert a (hugh size) ftp.retrbinary run into a
    >>> pseudo-file object with .read(bytes) method in order to not consume
    >>> 500MB on a copy operation.

    >
    >
    > [snip]
    >
    >>
    >>
    >> Hmmmm this is nearly there I think...:

    >
    >
    > whoops... spoke too soon..



    Trigger happy this morning...

    >
    >>
    >> import ftplib
    >>
    >> class TransferAbort(Exception): pass
    >>
    >> class FTPFile:
    >> def __init__(self, server, filename):
    >> self.server = server
    >> self.filename = filename
    >> self.offset = 0
    >>
    >> def callback(self, data):
    >> self.offset = self.offset + len(data)
    >> self.data = data
    >> ## now quit the RETR command?
    >> raise TransferAbort("stop right now")
    >>
    >> def read(self, amount):
    >> self.ftp = ftplib.FTP(self.server)
    >> self.ftp.login()

    >
    >
    >
    > I needed to insert a time.sleep(0.1) here as the connections were
    > falling over themselves - I guess testing with a blocksize of 24
    > is a little silly.
    >
    >
    >> try:
    >> self.ftp.retrbinary("RETR %s" %self.filename, self.callback,
    >> blocksize=amount,
    >> rest=self.offset)
    >> except TransferAbort:


    also need to close the ftp connection here!

    self.ftp.close()

    >> return self.data
    >>
    >>
    >> f = FTPFile("HOSTNAME", "FILENAME")
    >>
    >> print f.read(24)
    >> print f.read(24)
    >>

    >
    > ## new test...
    >
    > f = FTPFile("HOSTNAME", "FILENAME")
    >
    > while 1:
    > data = f.read(24)
    > if not data:
    > break
    > print data,
    >
    >
    >>
    >> I open the ftp connection inside the read method as it caused an error
    >> (on the second call to read) when I opened it in __init__ ???
    >>
    >> HTH
    >> Martin
    >>
    >>
     
    Martin Franklin, Feb 3, 2005
    #4
  5. Robert

    Robert Guest

    Re: convert ftp.retrbinary to file object? - Python language lacksexpression?

    That turns into periodic new RETR commands with offset. Think its more
    an "odd" trick. I'd even prefer a threaded approach (thread puts the
    blocks into a stack; a while ... yield generator loop in the main thread
    serves the .read() function of the pseudo file object, which is my
    wish). Yet such tricks are all kind of OS-level tricks with a lot of
    overhead.

    I wonder really, if the Python language itself can express an elegant
    flat solution to turn the block delivering callback function into a
    generator/.read(bytes) solution? I found no way.

    (Looking over some Ruby stuff, Ruby seems to be able to do so from the
    language. I am not really familiar to Ruby. I always felt Python to be
    as complete - but much more clean. I became somewhat jealous ... :) )

    As the solution in my case has to stand many different file systems
    compatibly ( file.read(bytes) function !) and also other FTPS & SFTP
    classes with different retrbinary functions have to be compatible, I
    cannot even make a simple FTP subclassed retrbinary without getting
    really weired. Thus the existing .retrbinary with callback is the
    "official interface in this game".



    > >> def callback(self, data):
    > >> self.offset = self.offset + len(data)
    > >> self.data = data
    > >> ## now quit the RETR command?
    > >> raise TransferAbort("stop right now")
     
    Robert, Feb 3, 2005
    #5
  6. Robert

    Steve Holden Guest

    Re: convert ftp.retrbinary to file object? - Python language lacksexpression?

    Robert wrote:

    > That turns into periodic new RETR commands with offset. Think its more
    > an "odd" trick. I'd even prefer a threaded approach (thread puts the
    > blocks into a stack; a while ... yield generator loop in the main thread
    > serves the .read() function of the pseudo file object, which is my
    > wish). Yet such tricks are all kind of OS-level tricks with a lot of
    > overhead.
    >
    > I wonder really, if the Python language itself can express an elegant
    > flat solution to turn the block delivering callback function into a
    > generator/.read(bytes) solution? I found no way.
    >

    Don't know whether this would be helpful as a starting point, but a
    while (hmm, some years ...) ago I wrote an example of how FTP could be
    used as a file-like object. Look for ftpStream.py on

    http://www.holdenweb.com/Python/

    Of course, in those days files could do a bit less than they can now, so
    there's no attempt to provide an iterator interface.

    > (Looking over some Ruby stuff, Ruby seems to be able to do so from the
    > language. I am not really familiar to Ruby. I always felt Python to be
    > as complete - but much more clean. I became somewhat jealous ... :) )
    >
    > As the solution in my case has to stand many different file systems
    > compatibly ( file.read(bytes) function !) and also other FTPS & SFTP
    > classes with different retrbinary functions have to be compatible, I
    > cannot even make a simple FTP subclassed retrbinary without getting
    > really weired. Thus the existing .retrbinary with callback is the
    > "official interface in this game".
    >

    You will note that my code uses delegation to an FTP object rather than
    inheritance. Maybe you would find that approach more fruitful for your
    application.

    regards
    Steve
    --
    Meet the Python developers and your c.l.py favorites March 23-25
    Come to PyCon DC 2005 http://www.pycon.org/
    Steve Holden http://www.holdenweb.com/
     
    Steve Holden, Feb 3, 2005
    #6
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. aum
    Replies:
    1
    Views:
    758
    Fredrik Lundh
    Jun 29, 2006
  2. Gabriel Genellina
    Replies:
    0
    Views:
    441
    Gabriel Genellina
    Jan 25, 2007
  3. Gabriel Genellina
    Replies:
    0
    Views:
    394
    Gabriel Genellina
    Jan 28, 2007
  4. Replies:
    3
    Views:
    5,274
  5. D. Buck
    Replies:
    2
    Views:
    566
    D. Buck
    Jun 29, 2004
Loading...

Share This Page