Memory error with quadratic interpolation

[Isaac Won]

Hi all,

I have tried to use different interpolation methods with Scipy. My code seems just fine with linear interpolation, but shows memory error with quadratic. I am a novice for python. I will appreciate any help.

#code
f = open(filin, "r")
for columns in ( raw.strip().split() for raw in f ):
a.append(columns[5])
x = np.array(a, float)


not_nan = np.logical_not(np.isnan(x))
indices = np.arange(len(x))
interp = interp1d(indices[not_nan], x[not_nan], kind = 'quadratic')

p = interp(indices)

------------------------------------------------------------------------
The number of data is 31747.

Thank you,

Isaac

 

[Ulrich Eckhardt]

Am 23.01.2013 05:06, schrieb Isaac Won:

I have tried to use different interpolation methods with Scipy. My
code seems just fine with linear interpolation, but shows memory
error with quadratic. I am a novice for python. I will appreciate any
help.

#code
f = open(filin, "r")

Check out the "with open(...) as f" syntax.

for columns in ( raw.strip().split() for raw in f ):

For the record, this first builds a sequence and then iterates over that
sequence. This is not very memory-efficient, try this instead:

for line in f:
columns = line.strip().split()


Concerning the rest of your problems, there is lots of code and the
datafile missing. However, there is also too much of it, try replacing
the file with generated data and remove everything from the code that is
not absolutely necessary.

Good luck!

Uli

 

[Oscar Benjamin]

Am 23.01.2013 05:06, schrieb Isaac Won:

I have tried to use different interpolation methods with Scipy. My
code seems just fine with linear interpolation, but shows memory
error with quadratic. I am a novice for python. I will appreciate any
help.

[SNIP]


Concerning the rest of your problems, there is lots of code and the datafile
missing. However, there is also too much of it, try replacing the file with
generated data and remove everything from the code that is not absolutely
necessary.

Also please copy paste the actual error message rather than paraphrasing it.


Oscar

 

[Isaac Won]

Hi all,



I have tried to use different interpolation methods with Scipy. My code seems just fine with linear interpolation, but shows memory error with quadratic. I am a novice for python. I will appreciate any help.



#code

f = open(filin, "r")

for columns in ( raw.strip().split() for raw in f ):

a.append(columns[5])

x = np.array(a, float)





not_nan = np.logical_not(np.isnan(x))

indices = np.arange(len(x))

interp = interp1d(indices[not_nan], x[not_nan], kind = 'quadratic')



p = interp(indices)



------------------------------------------------------------------------

The number of data is 31747.



Thank you,



Isaac

I really appreciate to both Ulich and Oscar.

To Oscar
My actual error message is:
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 311, in __init__
self._spline = splmake(x,oriented_y,order=order)
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 809, in splmake
coefs = func(xk, yk, order, conds, B)
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 530, in _find_smoothest
u,s,vh = np.dual.svd(B)
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/linalg/decomp_svd.py", line 91, in svd
full_matrices=full_matrices, overwrite_a = overwrite_a)
MemoryError

 

[Isaac Won]

Am 23.01.2013 05:06, schrieb Isaac Won:

I have tried to use different interpolation methods with Scipy. My
code seems just fine with linear interpolation, but shows memory
error with quadratic. I am a novice for python. I will appreciate any
help.

[SNIP]

Concerning the rest of your problems, there is lots of code and the datafile

missing. However, there is also too much of it, try replacing the file with

generated data and remove everything from the code that is not absolutely

necessary.

Also please copy paste the actual error message rather than paraphrasing it.





Oscar

I really appreciate to both Ulich and Oscar.

To Oscar
My actual error message is:
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 311, in __init__
self._spline = splmake(x,oriented_y,order=order)
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 809, in splmake
coefs = func(xk, yk, order, conds, B)
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 530, in _find_smoothest
u,s,vh = np.dual.svd(B)
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/linalg/decomp_svd.py", line 91, in svd
full_matrices=full_matrices, overwrite_a = overwrite_a)
MemoryError

 

[Isaac Won]

Am 23.01.2013 05:06, schrieb Isaac Won:

I have tried to use different interpolation methods with Scipy. My
code seems just fine with linear interpolation, but shows memory
error with quadratic. I am a novice for python. I will appreciate any
help.

[SNIP]

Concerning the rest of your problems, there is lots of code and the datafile

missing. However, there is also too much of it, try replacing the file with

generated data and remove everything from the code that is not absolutely

necessary.

Also please copy paste the actual error message rather than paraphrasing it.





Oscar

I really appreciate to both Ulich and Oscar.

To Oscar
My actual error message is:
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 311, in __init__
self._spline = splmake(x,oriented_y,order=order)
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 809, in splmake
coefs = func(xk, yk, order, conds, B)
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 530, in _find_smoothest
u,s,vh = np.dual.svd(B)
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/linalg/decomp_svd.py", line 91, in svd
full_matrices=full_matrices, overwrite_a = overwrite_a)
MemoryError

 

[Isaac Won]

On 23 January 2013 08:55, Ulrich Eckhardt

Am 23.01.2013 05:06, schrieb Isaac Won:

I have tried to use different interpolation methods with Scipy. My
code seems just fine with linear interpolation, but shows memory
error with quadratic. I am a novice for python. I will appreciate any
help.

[SNIP]

Concerning the rest of your problems, there is lots of code and the datafile

missing. However, there is also too much of it, try replacing the file with

generated data and remove everything from the code that is not absolutely

necessary.

Also please copy paste the actual error message rather than paraphrasing it.





Oscar

I really appreciate to both Ulich and Oscar.

To Oscar
My actual error message is:
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 311, in __init__
self._spline = splmake(x,oriented_y,order=order)
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 809, in splmake
coefs = func(xk, yk, order, conds, B)
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 530, in _find_smoothest
u,s,vh = np.dual.svd(B)
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/linalg/decomp_svd.py", line 91, in svd
full_matrices=full_matrices, overwrite_a = overwrite_a)
MemoryError

 

[Isaac Won]

On 23 January 2013 08:55, Ulrich Eckhardt

Am 23.01.2013 05:06, schrieb Isaac Won:

I have tried to use different interpolation methods with Scipy. My
code seems just fine with linear interpolation, but shows memory
error with quadratic. I am a novice for python. I will appreciate any
help.

[SNIP]

Concerning the rest of your problems, there is lots of code and the datafile

missing. However, there is also too much of it, try replacing the file with

generated data and remove everything from the code that is not absolutely

necessary.

Also please copy paste the actual error message rather than paraphrasing it.





Oscar

I really appreciate to both Ulich and Oscar.

To Oscar
My actual error message is:
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 311, in __init__
self._spline = splmake(x,oriented_y,order=order)
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 809, in splmake
coefs = func(xk, yk, order, conds, B)
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 530, in _find_smoothest
u,s,vh = np.dual.svd(B)
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/linalg/decomp_svd.py", line 91, in svd
full_matrices=full_matrices, overwrite_a = overwrite_a)
MemoryError

 

[Oscar Benjamin]

On Wednesday, January 23, 2013 4:08:13 AM UTC-6, Oscar Benjamin wrote:

To Oscar
My actual error message is:
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 311, in __init__
self._spline = splmake(x,oriented_y,order=order)
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 809, in splmake
coefs = func(xk, yk, order, conds, B)
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 530, in _find_smoothest
u,s,vh = np.dual.svd(B)
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/linalg/decomp_svd.py", line 91, in svd
full_matrices=full_matrices, overwrite_a = overwrite_a)
MemoryError

Are you sure that's the *whole* error message? The traceback only
refers to the scipy modules. I can't see the line from your code that
is generating the error.


Oscar

 

[Isaac Won]

Am 23.01.2013 05:06, schrieb Isaac Won:






Check out the "with open(...) as f" syntax.








For the record, this first builds a sequence and then iterates over that

sequence. This is not very memory-efficient, try this instead:



for line in f:

columns = line.strip().split()





Concerning the rest of your problems, there is lots of code and the

datafile missing. However, there is also too much of it, try replacing

the file with generated data and remove everything from the code that is

not absolutely necessary.



Good luck!



Uli

Hi Ulich,

I tried to change the code following your advice, but it doesn't seem to work still.

My adjusted code is:

a = []

with open(filin, "r") as f:

for line in f:
columns = line.strip().split()

a.append(columns[5])
x = np.array(a, float)


not_nan = np.logical_not(np.isnan(x))
indices = np.arange(len(x))
interp = interp1d(indices[not_nan], x[not_nan], kind = 'quadratic')
p = interp(indices)
---------------------------------------------------------------------
And full error message is:
interp = interp1d(indices[not_nan], x[not_nan], kind = 'quadratic')
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 311, in __init__
self._spline = splmake(x,oriented_y,order=order)
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 809, in splmake
coefs = func(xk, yk, order, conds, B)
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 530, in _find_smoothest
u,s,vh = np.dual.svd(B)
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/linalg/decomp_svd.py", line 91, in svd
full_matrices=full_matrices, overwrite_a = overwrite_a)
MemoryError
-----------------------------------------------------------------------
Could you give me some advice for this situation?

Thank you always,

Isaac

 

[Isaac Won]

Are you sure that's the *whole* error message? The traceback only

refers to the scipy modules. I can't see the line from your code that

is generating the error.





Oscar

Dear Oscar,

Following is full error message after I adjusted following Ulich's advice:

interp = interp1d(indices[not_nan], x[not_nan], kind = 'quadratic')
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 311, in __init__
self._spline = splmake(x,oriented_y,order=order)
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 809, in splmake
coefs = func(xk, yk, order, conds, B)
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 530, in _find_smoothest
u,s,vh = np.dual.svd(B)
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/linalg/decomp_svd.py", line 91, in svd
full_matrices=full_matrices, overwrite_a = overwrite_a)
MemoryError

 

[Isaac Won]

Are you sure that's the *whole* error message? The traceback only

refers to the scipy modules. I can't see the line from your code that

is generating the error.





Oscar

Dear Oscar,

Following is full error message after I adjusted following Ulich's advice:

interp = interp1d(indices[not_nan], x[not_nan], kind = 'quadratic')
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 311, in __init__
self._spline = splmake(x,oriented_y,order=order)
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 809, in splmake
coefs = func(xk, yk, order, conds, B)
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 530, in _find_smoothest
u,s,vh = np.dual.svd(B)
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/linalg/decomp_svd.py", line 91, in svd
full_matrices=full_matrices, overwrite_a = overwrite_a)
MemoryError

 

[Oscar Benjamin]

[SNIP]

Following is full error message after I adjusted following Ulich's advice:

interp = interp1d(indices[not_nan], x[not_nan], kind = 'quadratic')
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 311, in __init__
self._spline = splmake(x,oriented_y,order=order)
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 809, in splmake
coefs = func(xk, yk, order, conds, B)
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 530, in _find_smoothest
u,s,vh = np.dual.svd(B)
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/linalg/decomp_svd.py", line 91, in svd
full_matrices=full_matrices, overwrite_a = overwrite_a)
MemoryError

Where is the new code? You should show full working code (with the
import statements) and the full error that is generated by exactly
that code. If possible you should also write code that someone else
could run even without having access to your data files. If you did
that in your first post, you'd probably have an answer to your problem
by now.

Here is a version of your code that many people on this list can test
straight away:

import numpy as np
from scipy.interpolate import interp1d
x = np.array(31747 * [0.0], float)
indices = np.arange(len(x))
interp = interp1d(indices, x, kind='quadratic')

Running this gives the following error:

~$ python tmp.py
Traceback (most recent call last):
File "tmp.py", line 5, in <module>
interp = interp1d(indices, x, kind='quadratic')
File "/usr/lib/python2.7/dist-packages/scipy/interpolate/interpolate.py",
line 308, in __init__
self._spline = splmake(x,oriented_y,order=order)
File "/usr/lib/python2.7/dist-packages/scipy/interpolate/interpolate.py",
line 805, in splmake
B = _fitpack._bsplmat(order, xk)
MemoryError

Unless I've misunderstood how this function is supposed to be used, it
just doesn't really seem to work for arrays of much more than a few
hundred elements.


Oscar

 

[Isaac Won]

[SNIP]

Following is full error message after I adjusted following Ulich's advice:
interp = interp1d(indices[not_nan], x[not_nan], kind = 'quadratic')

File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 311, in __init__

self._spline = splmake(x,oriented_y,order=order)

File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 809, in splmake

coefs = func(xk, yk, order, conds, B)

File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 530, in _find_smoothest

u,s,vh = np.dual.svd(B)

File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/linalg/decomp_svd.py", line 91, in svd

full_matrices=full_matrices, overwrite_a = overwrite_a)

MemoryError

Where is the new code? You should show full working code (with the

import statements) and the full error that is generated by exactly

that code. If possible you should also write code that someone else

could run even without having access to your data files. If you did

that in your first post, you'd probably have an answer to your problem

by now.



Here is a version of your code that many people on this list can test

straight away:



import numpy as np

from scipy.interpolate import interp1d

x = np.array(31747 * [0.0], float)

indices = np.arange(len(x))

interp = interp1d(indices, x, kind='quadratic')



Running this gives the following error:



~$ python tmp.py

Traceback (most recent call last):

File "tmp.py", line 5, in <module>

interp = interp1d(indices, x, kind='quadratic')

File "/usr/lib/python2.7/dist-packages/scipy/interpolate/interpolate.py",

line 308, in __init__

self._spline = splmake(x,oriented_y,order=order)

File "/usr/lib/python2.7/dist-packages/scipy/interpolate/interpolate.py",

line 805, in splmake

B = _fitpack._bsplmat(order, xk)

MemoryError



Unless I've misunderstood how this function is supposed to be used, it

just doesn't really seem to work for arrays of much more than a few

hundred elements.





Oscar

Thank you Oscar for your help and advice.

I agree with you. So, I tried to find the way to solve this problem.

My full code adjusted is:
from scipy.interpolate import interp1d

import numpy as np
import matplotlib.pyplot as plt



with open(filin, "r") as f:

for line in f:
columns = line.strip().split()

a.append(columns[5])
x = np.array(a, float)


not_nan = np.logical_not(np.isnan(x))
indices = np.arange(len(x))
interp = interp1d(indices[not_nan], x[not_nan], kind = 'quadratic')

p = interp(indices)




k = np.arange(31747)

plt.subplot(211)
plt.plot(k, p)
plt.xlabel('Quadratic interpolation')
plt.subplot(212)
plt.plot(k, x)

plt.show()
-----------------------------------------------------------------
Whole error message was:

Traceback (most recent call last):
File "QI1.py", line 22, in <module>
interp = interp1d(indices[not_nan], x[not_nan], kind = 'quadratic')
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 311, in __init__
self._spline = splmake(x,oriented_y,order=order)
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 809, in splmake
coefs = func(xk, yk, order, conds, B)
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 530, in _find_smoothest
u,s,vh = np.dual.svd(B)
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/linalg/decomp_svd.py", line 91, in svd
full_matrices=full_matrices, overwrite_a = overwrite_a)
MemoryError

 

[Isaac Won]

[SNIP]

Following is full error message after I adjusted following Ulich's advice:
interp = interp1d(indices[not_nan], x[not_nan], kind = 'quadratic')

File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 311, in __init__

self._spline = splmake(x,oriented_y,order=order)

File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 809, in splmake

coefs = func(xk, yk, order, conds, B)

File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 530, in _find_smoothest

u,s,vh = np.dual.svd(B)

File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/linalg/decomp_svd.py", line 91, in svd

full_matrices=full_matrices, overwrite_a = overwrite_a)

MemoryError

Where is the new code? You should show full working code (with the

import statements) and the full error that is generated by exactly

that code. If possible you should also write code that someone else

could run even without having access to your data files. If you did

that in your first post, you'd probably have an answer to your problem

by now.



Here is a version of your code that many people on this list can test

straight away:



import numpy as np

from scipy.interpolate import interp1d

x = np.array(31747 * [0.0], float)

indices = np.arange(len(x))

interp = interp1d(indices, x, kind='quadratic')



Running this gives the following error:



~$ python tmp.py

Traceback (most recent call last):

File "tmp.py", line 5, in <module>

interp = interp1d(indices, x, kind='quadratic')

File "/usr/lib/python2.7/dist-packages/scipy/interpolate/interpolate.py",

line 308, in __init__

self._spline = splmake(x,oriented_y,order=order)

File "/usr/lib/python2.7/dist-packages/scipy/interpolate/interpolate.py",

line 805, in splmake

B = _fitpack._bsplmat(order, xk)

MemoryError



Unless I've misunderstood how this function is supposed to be used, it

just doesn't really seem to work for arrays of much more than a few

hundred elements.





Oscar

Thank you Oscar for your help and advice.

I agree with you. So, I tried to find the way to solve this problem.

My full code adjusted is:
from scipy.interpolate import interp1d

import numpy as np
import matplotlib.pyplot as plt



with open(filin, "r") as f:

for line in f:
columns = line.strip().split()

a.append(columns[5])
x = np.array(a, float)


not_nan = np.logical_not(np.isnan(x))
indices = np.arange(len(x))
interp = interp1d(indices[not_nan], x[not_nan], kind = 'quadratic')

p = interp(indices)




k = np.arange(31747)

plt.subplot(211)
plt.plot(k, p)
plt.xlabel('Quadratic interpolation')
plt.subplot(212)
plt.plot(k, x)

plt.show()
-----------------------------------------------------------------
Whole error message was:

Traceback (most recent call last):
File "QI1.py", line 22, in <module>
interp = interp1d(indices[not_nan], x[not_nan], kind = 'quadratic')
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 311, in __init__
self._spline = splmake(x,oriented_y,order=order)
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 809, in splmake
coefs = func(xk, yk, order, conds, B)
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 530, in _find_smoothest
u,s,vh = np.dual.svd(B)
File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/linalg/decomp_svd.py", line 91, in svd
full_matrices=full_matrices, overwrite_a = overwrite_a)
MemoryError

 

[Oscar Benjamin]

The solution is to use UnivariateSpline. I don't know what the
difference is but it works where the other fails:

import numpy as np
from scipy.interpolate import UnivariateSpline
x = np.array(10000 * [0.0], float)
indices = np.arange(len(x))
interp = UnivariateSpline(indices, x, k=2)
print(interp(1.5))


Oscar