binary input and memory address passing

[Eric Carlson]

Hello,

I can open, read, and convert data to a numeric (double) array from a
binary file using

nc = #something given
nr = #something given
f_o=open('junk.bin','rb')
x=reshape(array('d',f_o.read()),(nr,nc))

Is there a way in python that gives better performance? These commands
take three to 4 times longer than loading the data into an array in (for
example) octave.

Also, is there a way to pass the address of a block of memory, and then
access the data located there (pass by reference and let the python
program know what data type to use)?

My situation: I have two programs that can not communicate directly, but
I can get memory addresses for any data in my programs. I currently save
to a file, then load the data from file into my other program. Clearly
it would be much faster if I could access the block of memory directly
rather than making a copy.

Any suggestions greatly appreciated.

Cheers,

Eric Carlson

 

[Scott David Daniels]

Hello,

I can open, read, and convert data to a numeric (double) array from a
binary file using

nc = #something given
nr = #something given
f_o=open('junk.bin','rb')
x=reshape(array('d',f_o.read()),(nr,nc))

Is there a way in python that gives better performance? These commands
take three to 4 times longer than loading the data into an array in (for
example) octave.

Also, is there a way to pass the address of a block of memory, and then
access the data located there (pass by reference and let the python
program know what data type to use)?

My situation: I have two programs that can not communicate directly, but
I can get memory addresses for any data in my programs. I currently save
to a file, then load the data from file into my other program. Clearly
it would be much faster if I could access the block of memory directly
rather than making a copy.

Any suggestions greatly appreciated.

You might look over my (admittedly old)
http://members.dsl-only.net/~daniels/Block.html
It only has a Python 2.2 and 2.3 installer, but the sources are there
for you to compile (in which case you might toss a package back to me,
and I'll put it up). If you look it over and have questions, let me
know.

--Scott David Daniels
(e-mail address removed)

 

[Dennis Lee Bieber]

Also, is there a way to pass the address of a block of memory, and then
access the data located there (pass by reference and let the python
program know what data type to use)?

On a machine with both virtual memory, and process isolation, a
given "memory address" is specific to that process alone, and may be
physically different locations in memory (if they aren't different
locations it most likely means that one was swapped out to disk while
the other loaded over the memory). {This was both the weakness, and the
strength, of the Amiga OS -- having a non-virtual linear shared memory
space meant one process could corrupt others, but at the same time, one
could pass "messages" between programs by locating the named message
port of the receiver, and queuing a "message" (typically a structure
giving the address of the return message port, a length code, and the
address of the data buffer)}

My situation: I have two programs that can not communicate directly, but
I can get memory addresses for any data in my programs. I currently save
to a file, then load the data from file into my other program. Clearly
it would be much faster if I could access the block of memory directly
rather than making a copy.

The closest option I can think of currently would be to create a
named mmap that all programs open. Unfortunately, I don't know how easy
it is to actually map Python data to it (in a language with user
accessible pointers, one could define a large structure, and set the
pointer to the structure to the address /returned/ by a map).

--
Wulfraed Dennis Lee Bieber KD6MOG
(e-mail address removed) (e-mail address removed)
HTTP://wlfraed.home.netcom.com/
(Bestiaria Support Staff: (e-mail address removed))
HTTP://www.bestiaria.com/

 

[Eric Carlson]

Thanks Dennis and Scott for both responses. Since Dennis has slam dunked
my notion of sharing memory addresses (my ignorance in computing is
pretty much unbounded), I guess I will need to continue on with sharing
through files.

Opening up and loading the binary info into a string variable is very
fast. The process bogs when changing the representation to float. So it
appears that Scott's Block module might do what I need.

My original:

nc = #something given
nr = #something given
f_o=open('junk.bin','rb')
x=reshape(array('d',f_o.read()),(nr,nc))

gives me any array with nr rows and nc columns. Using Block I think I
would use:

f_o=open('junk.bin','rb')
x = View('d', Block(f_o.read()))

Okay, so now how can I use this? Is this like an array type so that

x = reshape(x,(nr,nc))

makes sense?

Regards,
Eric