access to preallocated block of memory?

Discussion in 'Python' started by Greg Copeland, Dec 14, 2005.

  1. I am running python on VxWorks. In the course of operation, a vxworks
    tasks writes to a reserved area of memory. I need access to this chunk
    of memory from within python. Initially I thought I could simply
    access it as a string but a string would reallocate and copy this chunk
    of memory; which is not something I can have as it would waste a huge
    amount of memory. We're talking about something like 40MB on a device
    with limited RAM. I have been looking at array. It looks promising.
    What's the best route to go here? Ideally, I would like to simply pass
    in the address of the reserved block and a length, and have the memory
    accessible.

    Is there some existing python object/facility I can use or will I need
    to create a custom module? Any tips, hints, or pointers would
    certainly be appreciated!


    Thanks,

    Greg
     
    Greg Copeland, Dec 14, 2005
    #1
    1. Advertising

  2. >>> vxworks tasks writes to a reserved area of memory.

    Is it compatible with the mmap module ?
     
    Do Re Mi chel La Si Do, Dec 15, 2005
    #2
    1. Advertising

  3. On 2005-12-15, Do Re Mi chel La Si Do <> wrote:
    >>>> vxworks tasks writes to a reserved area of memory.

    >
    > Is it compatible with the mmap module ?


    I've never written a VxWorks module, but it would only take a
    dozen or two lines of code to write a Linux driver that would
    impliment mmap() to allow a pre-defined block of address space
    to be mapped into user space.

    --
    Grant Edwards grante Yow! Yow! I threw up on
    at my window!
    visi.com
     
    Grant Edwards, Dec 15, 2005
    #3
  4. So array can not map a pre-existing chunk of memory? I did not port
    the mmap module because such semantics don't exist on VxWorks. Based
    on comments thus far, it looks like mmap is my best bet here? Any
    other options?
     
    Greg Copeland, Dec 15, 2005
    #4
  5. On 2005-12-15, Greg Copeland <> wrote:

    > So array can not map a pre-existing chunk of memory?


    Providing access to a pre-existing chunk of memory is an OS
    feature. How does VxWorks provide that feature?

    > I did not port the mmap module because such semantics don't
    > exist on VxWorks.


    Well then, what semantics _do_ exist on VxWorks?

    > Based on comments thus far, it looks like mmap is my best bet
    > here? Any other options?


    Since we don't know how VxWorks provides access to an arbitrary
    block of memory, how would we know how to use that VxWorks
    feature from Python?

    --
    Grant Edwards grante Yow! I'm DESPONDENT... I
    at hope there's something
    visi.com DEEP-FRIED under this
    miniature DOMED STADIUM...
     
    Grant Edwards, Dec 15, 2005
    #5
  6. On 14 Dec 2005 14:30:34 -0800, "Greg Copeland" <> wrote:

    >I am running python on VxWorks. In the course of operation, a vxworks
    >tasks writes to a reserved area of memory. I need access to this chunk
    >of memory from within python. Initially I thought I could simply
    >access it as a string but a string would reallocate and copy this chunk
    >of memory; which is not something I can have as it would waste a huge
    >amount of memory. We're talking about something like 40MB on a device
    >with limited RAM. I have been looking at array. It looks promising.
    >What's the best route to go here? Ideally, I would like to simply pass
    >in the address of the reserved block and a length, and have the memory
    >accessible.
    >
    >Is there some existing python object/facility I can use or will I need
    >to create a custom module? Any tips, hints, or pointers would
    >certainly be appreciated!


    What have you gathered from people who have gone before? googling python vxworks
    gives about 50k hits ;-)

    Your post does not have enough info about your environment, but for
    the sake of eliciting same, suppose you had a custom extension module
    written in C that would give you the access to the "reserved area of memory"
    that you want. So e.g. from the point of view of your python program, it looks
    like a module you can import, e.g.,

    import vxreservedmem

    Ok, how does the module know where the "reserved area" is? Would you link
    the C to some vx library interface to establish location and size? Or?
    Is there already a python interface to provide some access?
    Can there be more than one instance, so the module should be able to
    give you multiple objects that you can use to access different areas?

    Once you have an access-providing object, what kind of access do you require?
    What is represented within the "memory area" besides an array of bytes? Do the
    bytes represent C structs and primitive types? Are there access locks that
    determine when it's safe to touch the bytes? A single lock for the whole area,
    or individual locks for structs/subregions within the whole? Do you just need
    read access or do you want to store info? How would you like to select
    chunks of info? Just slices of byte arrays, or are there meaningful arrays
    of numbers -- integer, floats, etc. or bit fields etc?

    You could define a pure python vxresrvedmem module that just simulates the real
    thing, to test ideas -- and to communicate more clearly to us what the problem is.

    What modules/libraries do you have to give you access now from python to the vxworks
    environment? A file system? /dev/magic_stuff? or /proc/magic_stuff or ?


    Regards,
    Bengt Richter
     
    Bengt Richter, Dec 15, 2005
    #6
  7. Greg Copeland

    sam Guest

    I had a simular situation when I was writing Python routines to access
    the the storage (SCSIPASSTHROUGH) layer provided by Windows XP. My
    solution was to develope an extention in C that allocated all storage
    buffers within the extension. Then I added access extensions to
    controll reading and writing to and from these buffers. I would guess
    that you can write a vxworks extension that provides this linkage.
    Bengt Richter wrote:
    > On 14 Dec 2005 14:30:34 -0800, "Greg Copeland" <> wrote:
    >
    > >I am running python on VxWorks. In the course of operation, a vxworks
    > >tasks writes to a reserved area of memory. I need access to this chunk
    > >of memory from within python. Initially I thought I could simply
    > >access it as a string but a string would reallocate and copy this chunk
    > >of memory; which is not something I can have as it would waste a huge
    > >amount of memory. We're talking about something like 40MB on a device
    > >with limited RAM. I have been looking at array. It looks promising.
    > >What's the best route to go here? Ideally, I would like to simply pass
    > >in the address of the reserved block and a length, and have the memory
    > >accessible.
    > >
    > >Is there some existing python object/facility I can use or will I need
    > >to create a custom module? Any tips, hints, or pointers would
    > >certainly be appreciated!

    >
    > What have you gathered from people who have gone before? googling python vxworks
    > gives about 50k hits ;-)
    >
    > Your post does not have enough info about your environment, but for
    > the sake of eliciting same, suppose you had a custom extension module
    > written in C that would give you the access to the "reserved area of memory"
    > that you want. So e.g. from the point of view of your python program, it looks
    > like a module you can import, e.g.,
    >
    > import vxreservedmem
    >
    > Ok, how does the module know where the "reserved area" is? Would you link
    > the C to some vx library interface to establish location and size? Or?
    > Is there already a python interface to provide some access?
    > Can there be more than one instance, so the module should be able to
    > give you multiple objects that you can use to access different areas?
    >
    > Once you have an access-providing object, what kind of access do you require?
    > What is represented within the "memory area" besides an array of bytes? Do the
    > bytes represent C structs and primitive types? Are there access locks that
    > determine when it's safe to touch the bytes? A single lock for the whole area,
    > or individual locks for structs/subregions within the whole? Do you just need
    > read access or do you want to store info? How would you like to select
    > chunks of info? Just slices of byte arrays, or are there meaningful arrays
    > of numbers -- integer, floats, etc. or bit fields etc?
    >
    > You could define a pure python vxresrvedmem module that just simulates the real
    > thing, to test ideas -- and to communicate more clearly to us what the problem is.
    >
    > What modules/libraries do you have to give you access now from python to the vxworks
    > environment? A file system? /dev/magic_stuff? or /proc/magic_stuff or ?
    >
    >
    > Regards,
    > Bengt Richter
     
    sam, Dec 15, 2005
    #7
  8. First, let me say thanks for answering...

    > What have you gathered from people who have gone before? googling python vxworks

    gives about 50k hits

    And chances are, they will all be unrelated to my question. WRS uses
    python for various IDE scripting needs, but they do not use it on their
    own platform. Before I ported Python to VxWorks, AFAIK, it didn't
    exist on that platform.

    > our post does not have enough info about your environment,


    That's because I'm not asking someone to specifically tell me how to do
    it on VxWorks. This is because I'm sure a VxWorks specific solution
    does not exist. I'm asking about the best road to take given a generic
    python environment. I added the extra information just in case someone
    had some insight of the platform and python. If a generic answer does
    not exist then I'm sure I'll have to craft something my self...which is
    also part of what I'm trying to determine. The question is very
    simple...aside from mmap, does there exist any
    pre-existing facilities in python to which I can pass an address and an
    length and have the associated chunk of memory exposed to python while
    using an existing type, which does not attempt to malloc and copy?
    Unless you have a specific answer which uses vxWorks specific
    facilities, the fact that it's on VxWorks is strictly informational.

    > Once you have an access-providing object, what kind of access do you require?


    It doesn't really matter. It just so happens that it's read only
    access, but I can't see how it matters. Later on, it may require write
    access too. I simply need to access the binary data byte for byte.

    > What modules/libraries do you have to give you access now from python to the vxworks environment?


    As I said before, I have a chuck of memory, to which I have a pointer.
    I know the length of the memory. I would like to obtain acecss to the
    chuck of memory. Based on everything I'm seeing, it seems the answer
    is a custom module, all the way; which I can probably base the mmap
    module.

    Sorry for any confusion my original question may of created. I was
    rushed when I posted it. Hopefully this posting will be more clear as
    to my intentions and needs.

    Sincerely,

    Greg
     
    Greg Copeland, Dec 15, 2005
    #8
  9. I think you're getting caught in OS/platform semantics rather than a
    python solution. I already have access to the block on memory...I
    simply need information about existing python facilities which will
    allow me to expose the block to python as a native type...from which I
    can read byte for byte and optionally write to. As I originally said,
    I have a pointer and the block's length. I have access to it already.
    I simply need to expose it to python. What facilities already exist to
    allow this which do not cause malloc/memcpy on said block?

    Thus far, it looks like a hack on mmap is my best bet...I was hoping
    that the array module would have something I could use too; which would
    prevent me from havining to write my own data type module.
     
    Greg Copeland, Dec 15, 2005
    #9
  10. Based on the answers thus far, I suspect I'll being traveling this road
    shortly.

    Thanks,

    Greg
     
    Greg Copeland, Dec 15, 2005
    #10
  11. Greg Copeland

    Neil Hodgson Guest

    Greg Copeland:

    > I already have access to the block on memory...I
    > simply need information about existing python facilities which will
    > allow me to expose the block to python as a native type...from which I
    > can read byte for byte and optionally write to. As I originally said,
    > I have a pointer and the block's length. I have access to it already.
    > I simply need to expose it to python. What facilities already exist to
    > allow this which do not cause malloc/memcpy on said block?


    There is no generic array.at(address,length) in core Python as it
    would be a hole in Python's type system, allowing reading from or
    writing to any memory location and thus making it easier for Python to
    cause memory corruption and crashes. Libraries such as ctypes or its
    predecessor calldll do allow such access on some operating systems but
    not VxWorks. It would be quicker to write a special-purpose library for
    your job rather than port ctypes.

    Neil
     
    Neil Hodgson, Dec 15, 2005
    #11
  12. Dang it. That's what I suspected. Thanks!

    Greg
     
    Greg Copeland, Dec 15, 2005
    #12
  13. You might want to take a look at the "Blocks and Views" code I did,
    take a look at it and see if you can either use it directly or use
    it with any changes you feel like making.
    http://members.dsl-only.net/~daniels/Block.html
    --
    -Scott David Daniels
     
    Scott David Daniels, Dec 16, 2005
    #13
  14. That certainly looks interesting. I'll check it out right now.

    Thanks!

    Greg
     
    Greg Copeland, Dec 16, 2005
    #14
  15. What license does the code use? The PKG-INFO file says its MIT? This
    accurate? I'm still looking over the code, but it looks like I can do
    exactly what I need with only minor changes.
     
    Greg Copeland, Dec 16, 2005
    #15
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Fernando Lopes
    Replies:
    0
    Views:
    862
    Fernando Lopes
    Mar 29, 2005
  2. Alvin777
    Replies:
    2
    Views:
    435
    John Harrison
    Feb 21, 2004
  3. morrell
    Replies:
    1
    Views:
    967
    roy axenov
    Oct 10, 2006
  4. Replies:
    2
    Views:
    624
    joseph cook
    Jun 19, 2008
  5. Pinit Asavanuchit

    Is it possible to access memory as a block ?

    Pinit Asavanuchit, Aug 8, 2010, in forum: Ruby
    Replies:
    1
    Views:
    103
    Joel VanderWerf
    Aug 8, 2010
Loading...

Share This Page