API design for Python 2 / 3 compatibility

Discussion in 'Python' started by Stefan Schwarzer, Apr 13, 2013.

  1. Hello,

    I'm currently changing the FTP client library ftputil [1]
    so that the same code of the library works with Python
    2 (2.6 and up) and Python 3. (At the moment the code is for
    Python 2 only.) I've run into a API design issue where I
    don't know which API I should offer ftputil users under
    Python 2 and Python 3.

    [1] http://ftputil.sschwarzer.net/

    Some important background information: A key idea in ftputil
    is that it uses the same APIs as the Python standard library
    where possible. For example, with ftputil you can write code
    like this:

    with ftputil.FTPHost(host, user, password) as ftp_host:
    # Like `os.path.isdir`, but works on the FTP server.
    if ftp_host.path.isdir("hello_dir"):
    # Like `os.chdir`, but works on the FTP server.
    ftp_host.chdir("hello_dir")
    # Like the `open` builtin, but opens a remote file.
    with ftp_host.open("new_file", "w") as fobj:
    # Like `file.write` and `file.close`
    fobj.write("Hello world!")
    fobj.close()

    Since most of Python 2's and Python 3's filesystem-related
    APIs accept either bytes and character strings (and return
    the same type if they return anything string-like at all),
    the design here is rather clear to me.

    However, I have some difficulty with ftputil's counterpart
    of the `open` builtin function when files are opened for
    reading in text mode. Here are the approaches I've been
    thinking of so far:

    * Approach 1

    When opening remote text files for reading, ftputil will
    return byte strings from `read(line/s)` when run under
    Python 2 and unicode strings when run under Python 3.

    Pro: Each of the Python versions has ftputil behavior
    which matches the Python standard library behavior of
    the respective Python version.

    Con: Developers who want to use ftputil under Python 2
    _and_ 3 have to program against two different APIs since
    their code "inherits" ftputil's duality.

    Con: Support for two different APIs will make the
    ftputil code (at least a bit) more complicated than just
    returning unicode strings under both Python versions.

    * Approach 2

    When opening remote text files for reading, ftputil will
    always return unicode strings from `read(line/s)`,
    regardless of whether it runs under Python 2 or Python 3.

    Pro: Uniform API, independent on underlying Python
    version.

    Pro: Supporting a single API will result in cleaner code
    in ftputil than when supporting different APIs (see
    above).

    Con: This approach might break some code which expects
    the returned strings under Python 2 to be byte strings.

    Con: Developers who only use Python 2 might be confused
    if ftputil returns unicode strings from `read(line/s)`
    since this behavior doesn't match files opened with
    `open` in Python 2.

    Which approach do you recommend and why do you prefer that
    approach? Are there other approaches I have overlooked? Do
    you have other suggestions?

    Best regards,
    Stefan
    Stefan Schwarzer, Apr 13, 2013
    #1
    1. Advertising

  2. On 4/13/2013 12:36 PM, Stefan Schwarzer wrote:
    > Hello,
    >
    > I'm currently changing the FTP client library ftputil [1]
    > so that the same code of the library works with Python
    > 2 (2.6 and up) and Python 3. (At the moment the code is for
    > Python 2 only.) I've run into a API design issue where I
    > don't know which API I should offer ftputil users under
    > Python 2 and Python 3.
    >
    > [1] http://ftputil.sschwarzer.net/
    >
    > Some important background information: A key idea in ftputil
    > is that it uses the same APIs as the Python standard library
    > where possible. For example, with ftputil you can write code
    > like this:
    >
    > with ftputil.FTPHost(host, user, password) as ftp_host:
    > # Like `os.path.isdir`, but works on the FTP server.
    > if ftp_host.path.isdir("hello_dir"):
    > # Like `os.chdir`, but works on the FTP server.
    > ftp_host.chdir("hello_dir")
    > # Like the `open` builtin, but opens a remote file.
    > with ftp_host.open("new_file", "w") as fobj:
    > # Like `file.write` and `file.close`
    > fobj.write("Hello world!")
    > fobj.close()
    >
    > Since most of Python 2's and Python 3's filesystem-related
    > APIs accept either bytes and character strings (and return
    > the same type if they return anything string-like at all),
    > the design here is rather clear to me.
    >
    > However, I have some difficulty with ftputil's counterpart
    > of the `open` builtin function when files are opened for
    > reading in text mode. Here are the approaches I've been
    > thinking of so far:
    >
    > * Approach 1
    >
    > When opening remote text files for reading, ftputil will
    > return byte strings from `read(line/s)` when run under
    > Python 2 and unicode strings when run under Python 3.
    >
    > Pro: Each of the Python versions has ftputil behavior
    > which matches the Python standard library behavior of
    > the respective Python version.
    >
    > Con: Developers who want to use ftputil under Python 2
    > _and_ 3 have to program against two different APIs since
    > their code "inherits" ftputil's duality.
    >
    > Con: Support for two different APIs will make the
    > ftputil code (at least a bit) more complicated than just
    > returning unicode strings under both Python versions.
    >
    > * Approach 2
    >
    > When opening remote text files for reading, ftputil will
    > always return unicode strings from `read(line/s)`,
    > regardless of whether it runs under Python 2 or Python 3.
    >
    > Pro: Uniform API, independent on underlying Python
    > version.
    >
    > Pro: Supporting a single API will result in cleaner code
    > in ftputil than when supporting different APIs (see
    > above).
    >
    > Con: This approach might break some code which expects
    > the returned strings under Python 2 to be byte strings.
    >
    > Con: Developers who only use Python 2 might be confused
    > if ftputil returns unicode strings from `read(line/s)`
    > since this behavior doesn't match files opened with
    > `open` in Python 2.
    >
    > Which approach do you recommend and why do you prefer that
    > approach? Are there other approaches I have overlooked? Do
    > you have other suggestions?


    Approach 2 matches (or should match) io.open, which became builtin open
    in Python 3. I would simply document that ftp_host.open mimics io.open
    in the same way that ftp_host.chdir, etcetera, match os.chdir, etc. Your
    principle will remain intact.

    Anyone writing *new* Py 2 code with any idea of ever running on Py 3
    should be using io.open anyway. That is why it was backported. You might
    be able to reuse some io code or subclass some io classes for your
    implementation.
    Terry Jan Reedy, Apr 13, 2013
    #2
    1. Advertising

  3. Stefan Schwarzer

    Ethan Furman Guest

    On 04/13/2013 09:36 AM, Stefan Schwarzer wrote:
    > * Approach 2
    >
    > When opening remote text files for reading, ftputil will
    > always return unicode strings from `read(line/s)`,
    > regardless of whether it runs under Python 2 or Python 3.
    >
    > Pro: Uniform API, independent on underlying Python
    > version.
    >
    > Pro: Supporting a single API will result in cleaner code
    > in ftputil than when supporting different APIs (see
    > above).
    >
    > Con: This approach might break some code which expects
    > the returned strings under Python 2 to be byte strings.
    >
    > Con: Developers who only use Python 2 might be confused
    > if ftputil returns unicode strings from `read(line/s)`
    > since this behavior doesn't match files opened with
    > `open` in Python 2.
    >
    > Which approach do you recommend and why do you prefer that
    > approach?


    Approach 2, because it is much saner to deal with unicode inside the program, and only switch back to some kind of
    encoding when writing to files/pipes/etc. Since you are going to support python 3 as well you can bump the major
    version number and note the backward incompatibility.

    --
    ~Ethan~
    Ethan Furman, Apr 13, 2013
    #3
  4. Terry, Ethan:

    Thanks a lot for your excellent advice. :)

    On 2013-04-13 19:32, Terry Jan Reedy wrote:
    > Approach 2 matches (or should match) io.open, which became
    > builtin open in Python 3. I would simply document that
    > ftp_host.open mimics io.open in the same way that
    > ftp_host.chdir, etcetera, match os.chdir, etc. Your
    > principle will remain intact.


    I didn't know about `io.open` (or had forgotten it).

    > Anyone writing *new* Py 2 code with any idea of ever
    > running on Py 3 should be using io.open anyway. That is
    > why it was backported. You might be able to reuse some io
    > code or subclass some io classes for your implementation.


    Since I use `socket.makefile` to create the underlying file
    objects, I can use `BufferedReader`/`BufferedWriter` and
    `TextIOWrapper` to supply buffering and encoding/decoding.

    On 2013-04-13 20:03, Ethan Furman wrote:
    > Approach 2, because it is much saner to deal with unicode
    > inside the program, and only switch back to some kind of
    > encoding when writing to files/pipes/etc.


    Yes, this is a much saner design. I just was hesitant
    because of the introduced backward incompatibility and
    wanted to get other's opinions.

    > Since you are going to support python 3 as well you can
    > bump the major version number and note the backward
    > incompatibility.


    Actually I plan to increase the version number from 2.8 to
    3.0 because of the Python 3 support and already intend to
    change some module names that will be visible to client
    code. So this is also a good opportunity to clean up the
    file interface. :)

    Best regards,
    Stefan
    Stefan Schwarzer, Apr 14, 2013
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Praveen, Tayal (IE10)
    Replies:
    0
    Views:
    373
    Praveen, Tayal (IE10)
    Mar 17, 2005
  2. Mikhail T.
    Replies:
    6
    Views:
    326
    Mikhail T.
    May 14, 2007
  3. Robin Becker
    Replies:
    4
    Views:
    93
    Robin Becker
    Jan 17, 2014
  4. Chris Angelico
    Replies:
    0
    Views:
    63
    Chris Angelico
    Jan 17, 2014
  5. Chris Angelico
    Replies:
    0
    Views:
    78
    Chris Angelico
    Jan 17, 2014
Loading...

Share This Page