shared memory pointer

[Tim]

Hello Everyone,

I am getting shared memory in python using the following.

szName = c_char_p(name)
hMapObject = windll.kernel32.CreateFileMappingA(INVALID_HANDLE_VALUE,
None, PAGE_READONLY, 0, TABLE_SHMEMSIZE, szName)
if (hMapObject == 0):
print "OpenKey: Could not open name file mapping object"
raise WinError()

self.pData = windll.kernel32.MapViewOfFile(hMapObject,
FILE_MAP_ALL_ACCESS, 0, 0, TABLE_SHMEMSIZE)

This seems to work OK. How do I get to the data in self.pData? It is
suppose to be an array of floats. I think MapViewOfFile returns a
point to the data. Here is what I tried.

current_time = 45.55
memcpy( current_time, self.pData, 4 )

but I get an error:

ArgumentError: argument 1: <type 'exceptions.TypeError'>: Don't know
how to convert parameter 1

I need to get my floating point array casted from the self.pData. Any
help would be appreciated!

Thanks
Tim

 

[Tim Golden]

Hello Everyone,

I am getting shared memory in python using the following.

szName = c_char_p(name)
hMapObject = windll.kernel32.CreateFileMappingA(INVALID_HANDLE_VALUE,
None, PAGE_READONLY, 0, TABLE_SHMEMSIZE, szName)
if (hMapObject == 0):
print "OpenKey: Could not open name file mapping object"
raise WinError()

self.pData = windll.kernel32.MapViewOfFile(hMapObject,
FILE_MAP_ALL_ACCESS, 0, 0, TABLE_SHMEMSIZE)

Without answering your question directly, why not use
the stdlib mmap module, which does exactly this for
you behind the scenes?

Docs - http://docs.python.org/lib/module-mmap.html
Example - http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/413807

TJG

 

[Tim]

Without answering your question directly, why not use
the stdlib mmap module, which does exactly this for
you behind the scenes?

Docs -http://docs.python.org/lib/module-mmap.html
Example -http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/413807

TJG

I saw the mmap function in the shared memory example. I had some
concern with my large memory size being written to the disk drive. I
though it might slow down my application. The reason I am writting
this new code is because the existing method using a file. I thought
shared memory would be much faster.

I have tried using the memcpy and addresses, casting, etc. Is there a
way to use memcpy with the pData returned by MapViewOfFile()?

Thanks
Tim

 

[Tim]

Without answering your question directly, why not use
the stdlib mmap module, which does exactly this for
you behind the scenes?

Docs -http://docs.python.org/lib/module-mmap.html
Example -http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/413807

TJG

I reviewed the mmap function and I have a question. In the example
code below, what is the connection between the data in shared memory
and the mmap function. The fileno is zero. Why is it zero? The size
makes sense because there is 256 bytes in shared memory. The tag is
MyFileMappingObject. This tag does not have anything in common with
the shared memory. How can mmap point to any data in shared memory?
There is no coorelation.


pBuf = windll.kernel32.MapViewOfFile(hMapObject, FILE_MAP_ALL_ACCESS,
0, 0, SHMEMSIZE)
if (pBuf == 0):
print "Could not map view of file"
raise WinError()
else:
memcpy = cdll.msvcrt.memcpy
memcpy(pBuf, szMsg, len(szMsg))

shmem = mmap.mmap(0, 256, "MyFileMappingObject", mmap.ACCESS_WRITE)
shmem.write("Message from Python process")

 

[MC]

Hi!

I agree ; on windows mmap use Memory-Mapped-file, who use virtual
memory. And shared memory use physical memory.

The difference is OS an not Python

 

[Tim Golden]

I reviewed the mmap function and I have a question. In the example
code below, what is the connection between the data in shared memory
and the mmap function. The fileno is zero. Why is it zero? The size
makes sense because there is 256 bytes in shared memory. The tag is
MyFileMappingObject. This tag does not have anything in common with
the shared memory. How can mmap point to any data in shared memory?
There is no coorelation.


pBuf = windll.kernel32.MapViewOfFile(hMapObject, FILE_MAP_ALL_ACCESS,
0, 0, SHMEMSIZE)
if (pBuf == 0):
print "Could not map view of file"
raise WinError()
else:
memcpy = cdll.msvcrt.memcpy
memcpy(pBuf, szMsg, len(szMsg))

shmem = mmap.mmap(0, 256, "MyFileMappingObject", mmap.ACCESS_WRITE)
shmem.write("Message from Python process")

Your question seems a little strange. You appear to be creating
a memory mapping via ctypes and then expecting the Python mmap
module to know about it. Of course it doesn't. What you do is
to create two mmap mappings with the same tag name which --
under the covers -- point to the same shared memory. I just
tried it ad-hoc in two interpreters and it works fine.

I admit I am absolutely no expert on this area. My initial
suggestion was simply because your question triggered a
memory of what I'd seen elsewhere. Have a look at this
thread from a few years ago to see if it clarifies:

http://www.thescripts.com/forum/thread25421.html

(although what it -- and your question -- clearly do clarify
is a lack of clarity in the documentation!)

TJG

 

[Tim Golden]

I saw the mmap function in the shared memory example. I had some
concern with my large memory size being written to the disk drive. I
though it might slow down my application. The reason I am writting
this new code is because the existing method using a file. I thought
shared memory would be much faster.

I have tried using the memcpy and addresses, casting, etc. Is there a
way to use memcpy with the pData returned by MapViewOfFile()?

Tim, let's try to back up on this a moment. As I understand, you're
running under Windows and you want to use shared memory, presumably
between two processes. You know about the CreateFileMapping/MapViewOfFile
API, possibly from this example on MSDN [1] and you want to use that
technique in Windows *without* an explicit file backing.

Am I right so far?

I suggested the mmap module, and you seemed to pick up on it
and be trying to use both your ctypes solution *and* the mmap
module as two halves of the same mechanism. Maybe I misunderstood,
but that's what it looked like.

Then you asked a question about getting hold of a Python object's
memory address to be able to pass it into your ctypes solution.
(Which makes sense, given the nature of that solution).

What you seem to have missed is that your ctypes code is doing
*exactly* what the mmapmodule.c code (which is the implementation
of the mmap module) is doing for you behind the scenes.

Here's your code (very slightly reformatted):

<code>
szName = c_char_p(name)
hMapObject = windll.kernel32.CreateFileMappingA(
INVALID_HANDLE_VALUE,
None,
PAGE_READONLY,
0,
TABLE_SHMEMSIZE,
szName
)
if (hMapObject == 0):
print "OpenKey: Could not open name file mapping object"
raise WinError()

self.pData = windll.kernel32.MapViewOfFile(
hMapObject,
FILE_MAP_ALL_ACCESS,
0,
0,
TABLE_SHMEMSIZE
)

</code>

and here's the code from mmapmodule.c (also reformatted
and snipped about):

<code>
m_obj->map_handle = CreateFileMapping(
m_obj->file_handle,
NULL,
flProtect,
size_hi,
size_lo,
m_obj->tagname
);
if (m_obj->map_handle != NULL) {
m_obj->data = (char *) MapViewOfFile(
m_obj->map_handle,
dwDesiredAccess,
0,
0,
0
);

</code>

I hope you can see that they're basically doing the same
thing. (given the appropriate parameters). The only
clear difference is that the final param to MapViewOfFile
is 0 in the Python code, which the MSDN docs[2] indicate
"If this parameter is 0 (zero), the mapping extends
from the specified offset to the end of the file mapping."
It's not clear from that how it applies to a non-file-backed
FileMapping, but I *assume* that the Python devs have tested
that out.

In short, to have two Python processes talk via shared
memory (not linking to a specific file) the following
works for me:

<code-a>
import mmap

#
# The 0 special file value can be -1 in Python 2.5
#
shmem = mmap.mmap (0, 1000, "TJG", mmap.ACCESS_WRITE)
shmem.write ("blah blah")
</code-a>

<code-b>
import mmap

shmem = mmap.mmap (0, 1000, "TJG", mmap.ACCESS_WRITE)
print shmem.read (9)
</code>

Obviously, I've used dummy codes and values, but since
the .write and .read (and the other helper methods)
use strings, you can always pickle or marshal arbitrary
Python data to move it around.

I hope all that's helpful; if nothing else, it's given
me some exercise in reading the code of the stdlib, which
can't be bad!

TJG

[1] http://msdn2.microsoft.com/en-us/library/aa366551.aspx
[2] http://msdn2.microsoft.com/en-us/library/aa366761.aspx

 

[Tim]

Wow....You put some detail in this response. Thanks so much!

I think I want to stay away from mmap because it uses the disk to
store my memory. I am trying to stay away from that. I am building
strip charts in this python project to view my data. Currently, I am
using a file and I have to open the file when my simulation is
complete. The simulation takes too long to execute when it has to dump
all of the data to a file. The simulation is written in C++ but I want
the strip charts to be written in python so I can using the free plots
in matplotlib. I am trying to place the very large data block in
shared memory so I don't have to wait for the data to be written to a
disk.

I have about 200 shared memory segments that the simulation creates. I
did not know that my code would write to the hard drive because I use
ctypes. Could you explain that. Even if I use mmap, how could I mmap
200 shared memory segments? The first argument to mmap is a zero. How
does mmap know to map the returned object from MapViewOfFile to the
tag?

I have been looking into using a dll to do all of the mappings. Then,
I can call the dll from my python script to access my array of floats.
What do you think of this approach? I am having trouble finding good
documentation of this approach. Can you help?


Thanks

I saw the mmap function in the shared memory example. I had some
concern with my large memory size being written to the disk drive. I
though it might slow down my application. The reason I am writting
this new code is because the existing method using a file. I thought
shared memory would be much faster.

I have tried using the memcpy and addresses, casting, etc. Is there a
way to use memcpy with the pData returned by MapViewOfFile()?

Tim, let's try to back up on this a moment. As I understand, you're
running under Windows and you want to use shared memory, presumably
between two processes. You know about the CreateFileMapping/MapViewOfFile
API, possibly from this example on MSDN [1] and you want to use that
technique in Windows *without* an explicit file backing.

Am I right so far?

I suggested the mmap module, and you seemed to pick up on it
and be trying to use both your ctypes solution *and* the mmap
module as two halves of the same mechanism. Maybe I misunderstood,
but that's what it looked like.

Then you asked a question about getting hold of a Python object's
memory address to be able to pass it into your ctypes solution.
(Which makes sense, given the nature of that solution).

What you seem to have missed is that your ctypes code is doing
*exactly* what the mmapmodule.c code (which is the implementation
of the mmap module) is doing for you behind the scenes.

Here's your code (very slightly reformatted):

<code>
szName = c_char_p(name)
hMapObject = windll.kernel32.CreateFileMappingA(
INVALID_HANDLE_VALUE,
None,
PAGE_READONLY,
0,
TABLE_SHMEMSIZE,
szName
)
if (hMapObject == 0):
print "OpenKey: Could not open name file mapping object"
raise WinError()

self.pData = windll.kernel32.MapViewOfFile(
hMapObject,
FILE_MAP_ALL_ACCESS,
0,
0,
TABLE_SHMEMSIZE
)

</code>

and here's the code from mmapmodule.c (also reformatted
and snipped about):

<code>
m_obj->map_handle = CreateFileMapping(
m_obj->file_handle,
NULL,
flProtect,
size_hi,
size_lo,
m_obj->tagname
);
if (m_obj->map_handle != NULL) {
m_obj->data = (char *) MapViewOfFile(
m_obj->map_handle,
dwDesiredAccess,
0,
0,
0
);

</code>

I hope you can see that they're basically doing the same
thing. (given the appropriate parameters). The only
clear difference is that the final param to MapViewOfFile
is 0 in the Python code, which the MSDN docs[2] indicate
"If this parameter is 0 (zero), the mapping extends
from the specified offset to the end of the file mapping."
It's not clear from that how it applies to a non-file-backed
FileMapping, but I *assume* that the Python devs have tested
that out.

In short, to have two Python processes talk via shared
memory (not linking to a specific file) the following
works for me:

<code-a>
import mmap

#
# The 0 special file value can be -1 in Python 2.5
#
shmem = mmap.mmap (0, 1000, "TJG", mmap.ACCESS_WRITE)
shmem.write ("blah blah")
</code-a>

<code-b>
import mmap

shmem = mmap.mmap (0, 1000, "TJG", mmap.ACCESS_WRITE)
print shmem.read (9)
</code>

Obviously, I've used dummy codes and values, but since
the .write and .read (and the other helper methods)
use strings, you can always pickle or marshal arbitrary
Python data to move it around.

I hope all that's helpful; if nothing else, it's given
me some exercise in reading the code of the stdlib, which
can't be bad!

TJG

[1]http://msdn2.microsoft.com/en-us/library/aa366551.aspx
[2]http://msdn2.microsoft.com/en-us/library/aa366761.aspx

 

[Tim Golden]

I think I want to stay away from mmap because it uses the disk to
store my memory.

My point is that, whatever mmap is doing, your own code
is doing *exactly the same thing*. Passing the
INVALID_HANDLE_VALUE as both your code and the mmap
code are doing is documented as producing an area of
memory backed by the page file (which, let's face it,
is what all of your memory is backed by). Whether that's
good enough for you or not I don't know, but you're not
going to get any better with *this* technique

[... snip explanation of C++ simulator and Python charting ...]

OK, so your C++ stuff dumps to memory. Using CreateFileMapping
and MapViewOfFile? Or some other kind of global memory?

In short, if your C++ can write stuff out to shared memory
created using CreateFileMapping/MapViewOfFile with a first
param of INVALID_HANDLE_VALUE and a tag name, you can use
Python's mmap with the equivalent values and tag name to
access that data. Straightforward.

If, on the other hand, your C++ is writing to arbitrary
global memory, I don't think it's going to fly. You could
also use named pipes, by the way, and possibly other
Windows synching mechanisms. (He says, muddying the waters
horribly).

TJG