access to preallocated block of memory?

[Greg Copeland]

I am running python on VxWorks. In the course of operation, a vxworks
tasks writes to a reserved area of memory. I need access to this chunk
of memory from within python. Initially I thought I could simply
access it as a string but a string would reallocate and copy this chunk
of memory; which is not something I can have as it would waste a huge
amount of memory. We're talking about something like 40MB on a device
with limited RAM. I have been looking at array. It looks promising.
What's the best route to go here? Ideally, I would like to simply pass
in the address of the reserved block and a length, and have the memory
accessible.

Is there some existing python object/facility I can use or will I need
to create a custom module? Any tips, hints, or pointers would
certainly be appreciated!


Thanks,

Greg

 

[Do Re Mi chel La Si Do]

vxworks tasks writes to a reserved area of memory.

Is it compatible with the mmap module ?

 

[Grant Edwards]

Is it compatible with the mmap module ?

I've never written a VxWorks module, but it would only take a
dozen or two lines of code to write a Linux driver that would
impliment mmap() to allow a pre-defined block of address space
to be mapped into user space.

 

[Greg Copeland]

So array can not map a pre-existing chunk of memory? I did not port
the mmap module because such semantics don't exist on VxWorks. Based
on comments thus far, it looks like mmap is my best bet here? Any
other options?

 

[Grant Edwards]

So array can not map a pre-existing chunk of memory?

Providing access to a pre-existing chunk of memory is an OS
feature. How does VxWorks provide that feature?

I did not port the mmap module because such semantics don't
exist on VxWorks.

Well then, what semantics _do_ exist on VxWorks?

Based on comments thus far, it looks like mmap is my best bet
here? Any other options?

Since we don't know how VxWorks provides access to an arbitrary
block of memory, how would we know how to use that VxWorks
feature from Python?

 

[Bengt Richter]

I am running python on VxWorks. In the course of operation, a vxworks
tasks writes to a reserved area of memory. I need access to this chunk
of memory from within python. Initially I thought I could simply
access it as a string but a string would reallocate and copy this chunk
of memory; which is not something I can have as it would waste a huge
amount of memory. We're talking about something like 40MB on a device
with limited RAM. I have been looking at array. It looks promising.
What's the best route to go here? Ideally, I would like to simply pass
in the address of the reserved block and a length, and have the memory
accessible.

Is there some existing python object/facility I can use or will I need
to create a custom module? Any tips, hints, or pointers would
certainly be appreciated!

What have you gathered from people who have gone before? googling python vxworks
gives about 50k hits ;-)

Your post does not have enough info about your environment, but for
the sake of eliciting same, suppose you had a custom extension module
written in C that would give you the access to the "reserved area of memory"
that you want. So e.g. from the point of view of your python program, it looks
like a module you can import, e.g.,

import vxreservedmem

Ok, how does the module know where the "reserved area" is? Would you link
the C to some vx library interface to establish location and size? Or?
Is there already a python interface to provide some access?
Can there be more than one instance, so the module should be able to
give you multiple objects that you can use to access different areas?

Once you have an access-providing object, what kind of access do you require?
What is represented within the "memory area" besides an array of bytes? Do the
bytes represent C structs and primitive types? Are there access locks that
determine when it's safe to touch the bytes? A single lock for the whole area,
or individual locks for structs/subregions within the whole? Do you just need
read access or do you want to store info? How would you like to select
chunks of info? Just slices of byte arrays, or are there meaningful arrays
of numbers -- integer, floats, etc. or bit fields etc?

You could define a pure python vxresrvedmem module that just simulates the real
thing, to test ideas -- and to communicate more clearly to us what the problem is.

What modules/libraries do you have to give you access now from python to the vxworks
environment? A file system? /dev/magic_stuff? or /proc/magic_stuff or ?


Regards,
Bengt Richter

 

[sam]

I had a simular situation when I was writing Python routines to access
the the storage (SCSIPASSTHROUGH) layer provided by Windows XP. My
solution was to develope an extention in C that allocated all storage
buffers within the extension. Then I added access extensions to
controll reading and writing to and from these buffers. I would guess
that you can write a vxworks extension that provides this linkage.

 

[Greg Copeland]

First, let me say thanks for answering...

What have you gathered from people who have gone before? googling python vxworks

gives about 50k hits

And chances are, they will all be unrelated to my question. WRS uses
python for various IDE scripting needs, but they do not use it on their
own platform. Before I ported Python to VxWorks, AFAIK, it didn't
exist on that platform.

our post does not have enough info about your environment,

That's because I'm not asking someone to specifically tell me how to do
it on VxWorks. This is because I'm sure a VxWorks specific solution
does not exist. I'm asking about the best road to take given a generic
python environment. I added the extra information just in case someone
had some insight of the platform and python. If a generic answer does
not exist then I'm sure I'll have to craft something my self...which is
also part of what I'm trying to determine. The question is very
simple...aside from mmap, does there exist any
pre-existing facilities in python to which I can pass an address and an
length and have the associated chunk of memory exposed to python while
using an existing type, which does not attempt to malloc and copy?
Unless you have a specific answer which uses vxWorks specific
facilities, the fact that it's on VxWorks is strictly informational.

Once you have an access-providing object, what kind of access do you require?

It doesn't really matter. It just so happens that it's read only
access, but I can't see how it matters. Later on, it may require write
access too. I simply need to access the binary data byte for byte.

What modules/libraries do you have to give you access now from python to the vxworks environment?

As I said before, I have a chuck of memory, to which I have a pointer.
I know the length of the memory. I would like to obtain acecss to the
chuck of memory. Based on everything I'm seeing, it seems the answer
is a custom module, all the way; which I can probably base the mmap
module.

Sorry for any confusion my original question may of created. I was
rushed when I posted it. Hopefully this posting will be more clear as
to my intentions and needs.

Sincerely,

Greg

 

[Greg Copeland]

I think you're getting caught in OS/platform semantics rather than a
python solution. I already have access to the block on memory...I
simply need information about existing python facilities which will
allow me to expose the block to python as a native type...from which I
can read byte for byte and optionally write to. As I originally said,
I have a pointer and the block's length. I have access to it already.
I simply need to expose it to python. What facilities already exist to
allow this which do not cause malloc/memcpy on said block?

Thus far, it looks like a hack on mmap is my best bet...I was hoping
that the array module would have something I could use too; which would
prevent me from havining to write my own data type module.

 

[Greg Copeland]

Based on the answers thus far, I suspect I'll being traveling this road
shortly.

Thanks,

Greg

 

[Neil Hodgson]

Greg Copeland:

I already have access to the block on memory...I
simply need information about existing python facilities which will
allow me to expose the block to python as a native type...from which I
can read byte for byte and optionally write to. As I originally said,
I have a pointer and the block's length. I have access to it already.
I simply need to expose it to python. What facilities already exist to
allow this which do not cause malloc/memcpy on said block?

There is no generic array.at(address,length) in core Python as it
would be a hole in Python's type system, allowing reading from or
writing to any memory location and thus making it easier for Python to
cause memory corruption and crashes. Libraries such as ctypes or its
predecessor calldll do allow such access on some operating systems but
not VxWorks. It would be quicker to write a special-purpose library for
your job rather than port ctypes.

Neil

 

[Greg Copeland]

Dang it. That's what I suspected. Thanks!

Greg

 

[Scott David Daniels]

You might want to take a look at the "Blocks and Views" code I did,
take a look at it and see if you can either use it directly or use
it with any changes you feel like making.
http://members.dsl-only.net/~daniels/Block.html

 

[Greg Copeland]

That certainly looks interesting. I'll check it out right now.

Thanks!

Greg

 

[Greg Copeland]

What license does the code use? The PKG-INFO file says its MIT? This
accurate? I'm still looking over the code, but it looks like I can do
exactly what I need with only minor changes.