Re: How to receive a FILE* from Python under MinGW?

Discussion in 'Python' started by Gabriel Genellina, Mar 21, 2007.

  1. En Wed, 21 Mar 2007 00:46:03 -0300, John Pye
    <> escribió:

    > This is not an option for me, as I want to pass the
    > FILE* from Python and all the way into into a existing shared-library
    > code. I can't change the latter; it must work ok with standard fprintf
    > (etc) functions.


    You can get the file descriptor from the Python file object using its
    fileno() method. The file descriptor lives at the OS level, so it's safe
    to pass around. You can regenerate a new FILE struct (using the other
    runtime library) with fdopen.


    --
    Gabriel Genellina
    Gabriel Genellina, Mar 21, 2007
    #1
    1. Advertising

  2. Gabriel Genellina

    John Pye Guest

    On Mar 21, 3:15 pm, "Gabriel Genellina" <>
    wrote:
    > En Wed, 21 Mar 2007 00:46:03 -0300, John Pye
    > <> escribió:
    >
    > > This is not an option for me, as I want to pass the
    > > FILE* from Python and all the way into into a existing shared-library
    > > code. I can't change the latter; it must work ok with standard fprintf
    > > (etc) functions.

    >
    > You can get the file descriptor from the Python file object using its
    > fileno() method. The file descriptor lives at the OS level, so it's safe
    > to pass around. You can regenerate a new FILE struct (using the other
    > runtime library) with fdopen.
    >
    > --
    > Gabriel Genellina


    Hi Gabriel

    Are you sure about this? My understanding is that 'fileno' is just a
    FILE* cast to an integer. So it's a pointer to something (check out
    stdio.h to see what it points to). Problem is (AFAICT) that Python 2.4
    uses a different version of the C runtime DLL (MSVCRT*.DLL) to that
    which MinGW links against. And it turns out that the different C
    runtime libraries have incompatible implementations of the FILE
    struct. And therefore if you try to pass a FILE* (fileno?) from Python
    to MinGW you will get a segfault.

    If there is more to your suggestion that I'm not seeing, please
    clarify. Have you tried this with MinGW actually?

    Cheers
    JP
    John Pye, Mar 21, 2007
    #2
    1. Advertising

  3. Gabriel Genellina

    John Pye Guest

    On Mar 21, 3:15 pm, "Gabriel Genellina" <>
    wrote:
    > En Wed, 21 Mar 2007 00:46:03 -0300, John Pye
    > <> escribió:
    >
    > > This is not an option for me, as I want to pass the
    > > FILE* from Python and all the way into into a existing shared-library
    > > code. I can't change the latter; it must work ok with standard fprintf
    > > (etc) functions.

    >
    > You can get the file descriptor from the Python file object using its
    > fileno() method. The file descriptor lives at the OS level, so it's safe
    > to pass around. You can regenerate a new FILE struct (using the other
    > runtime library) with fdopen.
    >
    > --
    > Gabriel Genellina


    Hmm. Reading this again I think I understand properly now. The
    'fileno' is really *not* the same thing as the FILE* pointer cast to
    an integer. It is a lower-level structure.

    The question is now, if I don't want to modify my *python* code, how
    do I get at the 'fileno' property of my PyFileObject?

    Also, are there consequences for using this approach of regenerating
    the FILE struct? For example if I have the following:

    F = os.tmpfile()
    F.write("some data")
    F.seek(0)
    myswigmodule.dosomething(F)
    F.seek(0)
    S = F.read()
    print S

    I don't know enough about this low-level file handling to really
    understand the implications of what you are proposing -- perhaps you
    could explain a little?

    Cheers
    JP
    John Pye, Mar 21, 2007
    #3
  4. Gabriel Genellina

    Ross Ridge Guest

    Gabriel Genellina <> wrote:
    >You can get the file descriptor from the Python file object using its
    >fileno() method. The file descriptor lives at the OS level, so it's safe
    >to pass around.


    Not under Windows. Windows doesn't have Unix-like descriptors, so the
    C runtime emulates them.

    Ross Ridge

    --
    l/ // Ross Ridge -- The Great HTMU
    [oo][oo]
    -()-/()/ http://www.csclub.uwaterloo.ca/~rridge/
    db //
    Ross Ridge, Mar 21, 2007
    #4
  5. En Wed, 21 Mar 2007 01:30:34 -0300, John Pye <> escribió:

    > On Mar 21, 3:15 pm, "Gabriel Genellina" <>
    > wrote:
    >> En Wed, 21 Mar 2007 00:46:03 -0300, John Pye
    >> <> escribió:
    >>
    >> > This is not an option for me, as I want to pass the
    >> > FILE* from Python and all the way into into a existing shared-library
    >> > code. I can't change the latter; it must work ok with standard fprintf
    >> > (etc) functions.

    >>
    >> You can get the file descriptor from the Python file object using its
    >> fileno() method. The file descriptor lives at the OS level, so it's safe
    >> to pass around. You can regenerate a new FILE struct (using the other
    >> runtime library) with fdopen.
    >>
    >> --
    >> Gabriel Genellina

    >
    > Hi Gabriel
    >
    > Are you sure about this? My understanding is that 'fileno' is just a
    > FILE* cast to an integer. So it's a pointer to something (check out
    > stdio.h to see what it points to). Problem is (AFAICT) that Python 2.4


    A FILE struct usually *contains* a file descriptor, among other things.
    But I think the FILE struct is opaque and not specified. Perhaps one
    should go even at a lower level, using open_osfhandle and get_osfhandle.
    (From python you can use the msvcrt module). See
    http://msdn2.microsoft.com/en-us/library/kdfaxaay(VS.71).aspx
    h=get_osfhandle(f.fileno()) returns a Windows file handle that you can
    pass around.
    On the Mingw side: fd=open_osfhandle(h); FILE *f=fdopen(fd)

    > uses a different version of the C runtime DLL (MSVCRT*.DLL) to that
    > which MinGW links against. And it turns out that the different C
    > runtime libraries have incompatible implementations of the FILE
    > struct. And therefore if you try to pass a FILE* (fileno?) from Python
    > to MinGW you will get a segfault.


    Exactly; the idea is not to pass a foreign FILE struct, but to *contruct*
    another one based on an existing open file; this is what fdopen does.

    > If there is more to your suggestion that I'm not seeing, please
    > clarify. Have you tried this with MinGW actually?


    No, but I'm rather confident it should work... as always, it can fail
    miserably :)

    --
    Gabriel Genellina
    Gabriel Genellina, Mar 21, 2007
    #5
  6. Gabriel Genellina

    John Pye Guest

    On Mar 21, 4:04 pm, Ross Ridge <>
    wrote:
    > Gabriel Genellina <> wrote:
    > >You can get the file descriptor from the Python file object using its
    > >fileno() method. The file descriptor lives at the OS level, so it's safe
    > >to pass around.

    >
    > Not under Windows. Windows doesn't have Unix-like descriptors, so the
    > C runtime emulates them.


    I am trying the following... any thoughts?


    %typemap(in) FILE * {
    if (!PyFile_Check($input)) {
    PyErr_SetString(PyExc_TypeError, "Need a file!");
    return NULL;
    }
    %#ifdef __MINGW32__
    PyFileObject *fo = (PyFileObject *)$input;
    $1 = fdopen(_fileno(fo->f_fp),PyString_AsString(fo->f_mode));
    fprintf(stderr,"FDOPEN(%d,\"%s\")\n",fo->f_fp,PyString_AsString(fo-
    >f_mode));

    %#else
    $1 = PyFile_AsFile($input);
    %#endif
    }
    John Pye, Mar 21, 2007
    #6
  7. En Wed, 21 Mar 2007 02:04:30 -0300, Ross Ridge
    <> escribió:

    > Gabriel Genellina <> wrote:


    >> You can get the file descriptor from the Python file object using its
    >> fileno() method. The file descriptor lives at the OS level, so it's safe
    >> to pass around.

    >
    > Not under Windows. Windows doesn't have Unix-like descriptors, so the
    > C runtime emulates them.


    Using get_osfhandle on that pseudo-descriptor gives a Windows file handle;
    that handle should be equivalent (that is, it has a similar role:
    identifies an open file uniquely inside a process, and is independent on
    the C runtime library)

    --
    Gabriel Genellina
    Gabriel Genellina, Mar 21, 2007
    #7
  8. Gabriel Genellina

    John Pye Guest

    On Mar 21, 4:48 pm, "John Pye" <> wrote:
    >
    > I am trying the following... any thoughts?
    >
    > %typemap(in) FILE * {
    > if (!PyFile_Check($input)) {
    > PyErr_SetString(PyExc_TypeError, "Need a file!");
    > return NULL;
    > }
    > %#ifdef __MINGW32__
    > PyFileObject *fo = (PyFileObject *)$input;
    > $1 = fdopen(_fileno(fo->f_fp),PyString_AsString(fo->f_mode));
    > fprintf(stderr,"FDOPEN(%d,\"%s\")\n",fo->f_fp,PyString_AsString(fo->f_mode));
    >
    > %#else
    > $1 = PyFile_AsFile($input);
    > %#endif
    >
    > }



    Ok, so this didn't work. 'fdopen' returned NULL. I guess that nails
    the coffin for the fileno() approach?

    Cheers
    JP
    John Pye, Mar 21, 2007
    #8
  9. En Wed, 21 Mar 2007 01:58:05 -0300, John Pye <> escribió:

    > Hmm. Reading this again I think I understand properly now. The
    > 'fileno' is really *not* the same thing as the FILE* pointer cast to
    > an integer. It is a lower-level structure.

    Just an integer. But as Ross Ridge has pointed out, it's not an OS thing
    on Windows, it's faked by the C runtime.

    > The question is now, if I don't want to modify my *python* code, how
    > do I get at the 'fileno' property of my PyFileObject?


    From Python, calling the fileno() method. From C, using
    fileno(PyFile_AsFile(your_file_object))
    And then, using get_osfhandle(fd)

    > Also, are there consequences for using this approach of regenerating
    > the FILE struct? For example if I have the following:
    >
    > F = os.tmpfile()
    > F.write("some data")
    > F.seek(0)
    > myswigmodule.dosomething(F)
    > F.seek(0)
    > S = F.read()
    > print S


    Just try it and let us know what happens! :)

    > I don't know enough about this low-level file handling to really
    > understand the implications of what you are proposing -- perhaps you
    > could explain a little?


    It's too late (for me) now, but if you can't make it work I'll try an
    example tomorrow.

    --
    Gabriel Genellina
    Gabriel Genellina, Mar 21, 2007
    #9
  10. Gabriel Genellina

    John Pye Guest

    Gabriel, if you think you can make an example that works, that would
    be great. I'm afraid I feel a bit out of my depth and don't have much
    confidence in this idea.

    JP
    John Pye, Mar 21, 2007
    #10
  11. En Wed, 21 Mar 2007 03:35:09 -0300, John Pye <> escribió:

    > Gabriel, if you think you can make an example that works, that would
    > be great. I'm afraid I feel a bit out of my depth and don't have much
    > confidence in this idea.


    Try this. It's based on the example_nt extension module.

    --- begin example.c ---
    #include <io.h>
    #include <fcntl.h>
    #include <stdio.h>
    #include <windows.h>
    #include "Python.h"

    static void externalfunction(FILE* f)
    {
    // This is a "fake" external function that receives a FILE*
    printf("In externalfunction\n");
    fprintf(f, "Hello from externalfunction!\n");
    }

    static void wrapper(DWORD osfhandle)
    {
    // This is a wrapper around externalfunction; receives an OS handle
    // for an open file, builds a FILE structure, and calls externalfunction
    FILETIME lw;
    SYSTEMTIME st;
    FILE* f;

    // This call is just to show that osfhandle is actually a Windows handle
    // as if one had used CreateFile(...) by example
    printf("Using osfhandle with GetFileTime\n");
    GetFileTime((HANDLE)osfhandle, NULL, NULL, &lw);
    FileTimeToSystemTime(&lw, &st);
    printf("LastWrite %d-%d-%d %02d:%02d:%02d.%03d\n", st.wYear, st.wMonth,
    st.wDay, st.wHour, st.wMinute, st.wSecond, st.wMilliseconds);

    // Now build a FILE struct from the received handle
    f = fdopen(_open_osfhandle(osfhandle,_O_APPEND),"a");
    externalfunction(f);
    fflush(f); // ensure all buffers are written
    }

    static PyObject *
    filetest(PyObject *self, PyObject *args)
    {
    DWORD handle;

    if (!PyArg_ParseTuple(args, "I", &handle))
    return NULL;
    printf("Received handle: %d\n", handle);
    wrapper(handle);
    printf("Done\n");
    Py_INCREF(Py_None);
    return Py_None;
    }

    static PyMethodDef example_methods[] = {
    {"filetest", filetest, 1, "filetest doc string"},
    {NULL, NULL}
    };

    void
    initexample(void)
    {
    Py_InitModule("example", example_methods);
    }
    --- end example.c ---

    --- begin test.py ---
    f = open("output.txt","w")
    f.write("Hello from python!\n")
    # ensure all buffers are written before calling the external function
    f.flush()

    import msvcrt
    fh = msvcrt.get_osfhandle(f.fileno())
    print "handle=",fh

    import example
    # calling example.filetest using the OS handle
    example.filetest(fh)
    f.close()

    f = open("output.txt","r")
    print "output.txt:"
    print f.read()
    f.close()
    --- test.py ---

    Build example.dll and rename it example.pyd; copy test.py to the same
    directory.
    Running test.py gives:

    C:\APPS\Python24\PYTHON~1.2\TEST_M~1\Release>python24 test.py
    handle= 1988
    Received handle: 1988
    Using osfhandle with GetFileTime
    LastWrite 2007-3-23 04:25:08.000
    In externalfunction
    Done
    output.txt:
    Hello from python!
    Hello from externalfunction!

    --
    Gabriel Genellina
    Gabriel Genellina, Mar 23, 2007
    #11
  12. Gabriel Genellina

    John Pye Guest

    On Mar 23, 3:33 pm, "Gabriel Genellina" <>
    wrote:
    > import msvcrt
    > fh = msvcrt.get_osfhandle(f.fileno())

    ...
    > example.filetest(fh)
    > f.close()


    Cool, that looks great, Gabriel.

    But is there any way I can hide the get_osfhandle call inside my
    Python module? That way I wouldn't need to request end users to make
    contorted 'if platform.system()=="Windows"' calls everywhere.

    Maybe this *is* workable, after all :)
    John Pye, Mar 23, 2007
    #12
  13. En Fri, 23 Mar 2007 01:47:22 -0300, John Pye <> escribió:

    > On Mar 23, 3:33 pm, "Gabriel Genellina" <>
    > wrote:
    >> import msvcrt
    >> fh = msvcrt.get_osfhandle(f.fileno())

    > ..
    >> example.filetest(fh)
    >> f.close()

    >
    > Cool, that looks great, Gabriel.
    >
    > But is there any way I can hide the get_osfhandle call inside my
    > Python module? That way I wouldn't need to request end users to make
    > contorted 'if platform.system()=="Windows"' calls everywhere.
    >
    > Maybe this *is* workable, after all :)


    I can think of two ways:

    - Define a Python function to do that, and test the platform just there.
    Something like that:

    def exportable_file(f):
    if we_are_working_on_windows:
    import msvcrt
    return msvcrt.get_osfhandle(f.fileno())
    else:
    return f

    And replace all places where a Python file goes into a C extension, with
    exportable_file(f)

    - Define your own file class, inheriting from file, and store that handle
    as an attribute

    class my_file_class(file):
    def __init__(self, name, mode="r", buffering=-1):
    file.__init__(self, name, mode, buffering)
    self.handle = msvcrt.get_osfhandle(self.fileno())

    But I've not checked this; I'm not sure if file creation always goes thru
    this __init__; I assume the file will never change its FILE struct (f_fp
    member) (uhm, is there any way to re-open a file?). Perhaps this approach
    has many assumptions to be usable at all - uh, forget it.


    --
    Gabriel Genellina
    Gabriel Genellina, Mar 23, 2007
    #13
  14. Gabriel Genellina

    John Pye Guest

    On Mar 23, 7:48 pm, "Gabriel Genellina" <>
    wrote:
    >
    > And replace all places where a Python file goes into a C extension, with
    > exportable_file(f)


    What about calling mscvrt_get_osfhandle from inside the SWIG wrapper?
    I tried this but it seemed that the function was not exported to the
    DLL.

    Cheers
    JP
    John Pye, Mar 23, 2007
    #14
  15. En Fri, 23 Mar 2007 10:07:30 -0300, John Pye <> escribió:

    > On Mar 23, 7:48 pm, "Gabriel Genellina" <>
    > wrote:
    >>
    >> And replace all places where a Python file goes into a C extension, with
    >> exportable_file(f)

    >
    > What about calling mscvrt_get_osfhandle from inside the SWIG wrapper?
    > I tried this but it seemed that the function was not exported to the
    > DLL.


    The idea is to separate both worlds - _get_osfhandle must be called from
    the C runtime used by Python, not the one linked to your extension.

    --
    Gabriel Genellina
    Gabriel Genellina, Mar 23, 2007
    #15
  16. En Fri, 23 Mar 2007 16:34:22 -0300, Gabriel Genellina
    <> escribió:

    >> What about calling mscvrt_get_osfhandle from inside the SWIG wrapper?
    >> I tried this but it seemed that the function was not exported to the
    >> DLL.

    >
    > The idea is to separate both worlds - _get_osfhandle must be called from
    > the C runtime used by Python, not the one linked to your extension.


    Ah, but you could import the msvcrt module and get it from there. *ONLY*
    get_osfhandle. The other function, open_osfhandle, must be from the
    runtime used by your extension.

    --
    Gabriel Genellina
    Gabriel Genellina, Mar 23, 2007
    #16
  17. Gabriel Genellina

    John Pye Guest

    On Mar 24, 5:37 am, "Gabriel Genellina" <>
    wrote:
    > En Fri, 23 Mar 2007 16:34:22 -0300, Gabriel Genellina
    > <> escribió:
    >
    > >> What about calling mscvrt_get_osfhandle from inside the SWIG wrapper?
    > >> I tried this but it seemed that the function was not exported to the
    > >> DLL.

    >
    > > The idea is to separate both worlds - _get_osfhandle must be called from
    > > the C runtime used by Python, not the one linked to your extension.

    >
    > Ah, but you could import the msvcrt module and get it from there. *ONLY*
    > get_osfhandle. The other function, open_osfhandle, must be from the
    > runtime used by your extension.
    >
    > --
    > Gabriel Genellina


    Yes, that was what I was hoping might be possible. Have not had any
    luck with that yet though.

    Cheers
    JP
    John Pye, Mar 25, 2007
    #17
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Sam Smith
    Replies:
    0
    Views:
    1,159
    Sam Smith
    Oct 18, 2004
  2. John Pye
    Replies:
    0
    Views:
    448
    John Pye
    Mar 21, 2007
  3. Carl Douglas
    Replies:
    11
    Views:
    512
    Robert Kern
    Mar 23, 2007
  4. xeno fears
    Replies:
    1
    Views:
    493
    Martin v. Löwis
    Oct 4, 2009
  5. Stefan Ram
    Replies:
    5
    Views:
    129
    Stefan Ram
    Mar 17, 2014
Loading...

Share This Page