Memory error with quadratic interpolation

Discussion in 'Python' started by Isaac Won, Jan 23, 2013.

  1. Isaac Won

    Isaac Won Guest

    Hi all,

    I have tried to use different interpolation methods with Scipy. My code seems just fine with linear interpolation, but shows memory error with quadratic. I am a novice for python. I will appreciate any help.

    #code
    f = open(filin, "r")
    for columns in ( raw.strip().split() for raw in f ):
    a.append(columns[5])
    x = np.array(a, float)


    not_nan = np.logical_not(np.isnan(x))
    indices = np.arange(len(x))
    interp = interp1d(indices[not_nan], x[not_nan], kind = 'quadratic')

    p = interp(indices)

    ------------------------------------------------------------------------
    The number of data is 31747.

    Thank you,

    Isaac
    Isaac Won, Jan 23, 2013
    #1
    1. Advertising

  2. Am 23.01.2013 05:06, schrieb Isaac Won:
    > I have tried to use different interpolation methods with Scipy. My
    > code seems just fine with linear interpolation, but shows memory
    > error with quadratic. I am a novice for python. I will appreciate any
    > help.
    >
    > #code
    > f = open(filin, "r")


    Check out the "with open(...) as f" syntax.


    > for columns in ( raw.strip().split() for raw in f ):


    For the record, this first builds a sequence and then iterates over that
    sequence. This is not very memory-efficient, try this instead:

    for line in f:
    columns = line.strip().split()


    Concerning the rest of your problems, there is lots of code and the
    datafile missing. However, there is also too much of it, try replacing
    the file with generated data and remove everything from the code that is
    not absolutely necessary.

    Good luck!

    Uli
    Ulrich Eckhardt, Jan 23, 2013
    #2
    1. Advertising

  3. On 23 January 2013 08:55, Ulrich Eckhardt
    <> wrote:
    > Am 23.01.2013 05:06, schrieb Isaac Won:
    >
    >> I have tried to use different interpolation methods with Scipy. My
    >> code seems just fine with linear interpolation, but shows memory
    >> error with quadratic. I am a novice for python. I will appreciate any
    >> help.

    >

    [SNIP]
    >
    >
    > Concerning the rest of your problems, there is lots of code and the datafile
    > missing. However, there is also too much of it, try replacing the file with
    > generated data and remove everything from the code that is not absolutely
    > necessary.


    Also please copy paste the actual error message rather than paraphrasing it.


    Oscar
    Oscar Benjamin, Jan 23, 2013
    #3
  4. Isaac Won

    Isaac Won Guest

    On Tuesday, January 22, 2013 10:06:41 PM UTC-6, Isaac Won wrote:
    > Hi all,
    >
    >
    >
    > I have tried to use different interpolation methods with Scipy. My code seems just fine with linear interpolation, but shows memory error with quadratic. I am a novice for python. I will appreciate any help.
    >
    >
    >
    > #code
    >
    > f = open(filin, "r")
    >
    > for columns in ( raw.strip().split() for raw in f ):
    >
    > a.append(columns[5])
    >
    > x = np.array(a, float)
    >
    >
    >
    >
    >
    > not_nan = np.logical_not(np.isnan(x))
    >
    > indices = np.arange(len(x))
    >
    > interp = interp1d(indices[not_nan], x[not_nan], kind = 'quadratic')
    >
    >
    >
    > p = interp(indices)
    >
    >
    >
    > ------------------------------------------------------------------------
    >
    > The number of data is 31747.
    >
    >
    >
    > Thank you,
    >
    >
    >
    > Isaac


    I really appreciate to both Ulich and Oscar.

    To Oscar
    My actual error message is:
    File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 311, in __init__
    self._spline = splmake(x,oriented_y,order=order)
    File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 809, in splmake
    coefs = func(xk, yk, order, conds, B)
    File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 530, in _find_smoothest
    u,s,vh = np.dual.svd(B)
    File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/linalg/decomp_svd.py", line 91, in svd
    full_matrices=full_matrices, overwrite_a = overwrite_a)
    MemoryError
    --------------------------------------------------------------------------
    Thank you,

    Hoonill
    Isaac Won, Jan 23, 2013
    #4
  5. Isaac Won

    Isaac Won Guest

    On Wednesday, January 23, 2013 4:08:13 AM UTC-6, Oscar Benjamin wrote:
    > On 23 January 2013 08:55, Ulrich Eckhardt
    >
    > <> wrote:
    >
    > > Am 23.01.2013 05:06, schrieb Isaac Won:

    >
    > >

    >
    > >> I have tried to use different interpolation methods with Scipy. My

    >
    > >> code seems just fine with linear interpolation, but shows memory

    >
    > >> error with quadratic. I am a novice for python. I will appreciate any

    >
    > >> help.

    >
    > >

    >
    > [SNIP]
    >
    > >

    >
    > >

    >
    > > Concerning the rest of your problems, there is lots of code and the datafile

    >
    > > missing. However, there is also too much of it, try replacing the file with

    >
    > > generated data and remove everything from the code that is not absolutely

    >
    > > necessary.

    >
    >
    >
    > Also please copy paste the actual error message rather than paraphrasing it.
    >
    >
    >
    >
    >
    > Oscar


    I really appreciate to both Ulich and Oscar.

    To Oscar
    My actual error message is:
    File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 311, in __init__
    self._spline = splmake(x,oriented_y,order=order)
    File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 809, in splmake
    coefs = func(xk, yk, order, conds, B)
    File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 530, in _find_smoothest
    u,s,vh = np.dual.svd(B)
    File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/linalg/decomp_svd.py", line 91, in svd
    full_matrices=full_matrices, overwrite_a = overwrite_a)
    MemoryError
    --------------------------------------------------------------------------
    Thank you,

    Isaac
    Isaac Won, Jan 23, 2013
    #5
  6. Isaac Won

    Isaac Won Guest

    On Wednesday, January 23, 2013 4:08:13 AM UTC-6, Oscar Benjamin wrote:
    > On 23 January 2013 08:55, Ulrich Eckhardt
    >
    > <> wrote:
    >
    > > Am 23.01.2013 05:06, schrieb Isaac Won:

    >
    > >

    >
    > >> I have tried to use different interpolation methods with Scipy. My

    >
    > >> code seems just fine with linear interpolation, but shows memory

    >
    > >> error with quadratic. I am a novice for python. I will appreciate any

    >
    > >> help.

    >
    > >

    >
    > [SNIP]
    >
    > >

    >
    > >

    >
    > > Concerning the rest of your problems, there is lots of code and the datafile

    >
    > > missing. However, there is also too much of it, try replacing the file with

    >
    > > generated data and remove everything from the code that is not absolutely

    >
    > > necessary.

    >
    >
    >
    > Also please copy paste the actual error message rather than paraphrasing it.
    >
    >
    >
    >
    >
    > Oscar


    I really appreciate to both Ulich and Oscar.

    To Oscar
    My actual error message is:
    File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 311, in __init__
    self._spline = splmake(x,oriented_y,order=order)
    File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 809, in splmake
    coefs = func(xk, yk, order, conds, B)
    File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 530, in _find_smoothest
    u,s,vh = np.dual.svd(B)
    File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/linalg/decomp_svd.py", line 91, in svd
    full_matrices=full_matrices, overwrite_a = overwrite_a)
    MemoryError
    --------------------------------------------------------------------------
    Thank you,

    Isaac
    Isaac Won, Jan 23, 2013
    #6
  7. Isaac Won

    Isaac Won Guest

    On Wednesday, January 23, 2013 4:08:13 AM UTC-6, Oscar Benjamin wrote:
    > On 23 January 2013 08:55, Ulrich Eckhardt
    >
    >
    >
    > > Am 23.01.2013 05:06, schrieb Isaac Won:

    >
    > >

    >
    > >> I have tried to use different interpolation methods with Scipy. My

    >
    > >> code seems just fine with linear interpolation, but shows memory

    >
    > >> error with quadratic. I am a novice for python. I will appreciate any

    >
    > >> help.

    >
    > >

    >
    > [SNIP]
    >
    > >

    >
    > >

    >
    > > Concerning the rest of your problems, there is lots of code and the datafile

    >
    > > missing. However, there is also too much of it, try replacing the file with

    >
    > > generated data and remove everything from the code that is not absolutely

    >
    > > necessary.

    >
    >
    >
    > Also please copy paste the actual error message rather than paraphrasing it.
    >
    >
    >
    >
    >
    > Oscar


    I really appreciate to both Ulich and Oscar.

    To Oscar
    My actual error message is:
    File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 311, in __init__
    self._spline = splmake(x,oriented_y,order=order)
    File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 809, in splmake
    coefs = func(xk, yk, order, conds, B)
    File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 530, in _find_smoothest
    u,s,vh = np.dual.svd(B)
    File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/linalg/decomp_svd.py", line 91, in svd
    full_matrices=full_matrices, overwrite_a = overwrite_a)
    MemoryError
    --------------------------------------------------------------------------
    Thank you,

    Hoonill
    Isaac Won, Jan 23, 2013
    #7
  8. Isaac Won

    Isaac Won Guest

    On Wednesday, January 23, 2013 4:08:13 AM UTC-6, Oscar Benjamin wrote:
    > On 23 January 2013 08:55, Ulrich Eckhardt
    >
    >
    >
    > > Am 23.01.2013 05:06, schrieb Isaac Won:

    >
    > >

    >
    > >> I have tried to use different interpolation methods with Scipy. My

    >
    > >> code seems just fine with linear interpolation, but shows memory

    >
    > >> error with quadratic. I am a novice for python. I will appreciate any

    >
    > >> help.

    >
    > >

    >
    > [SNIP]
    >
    > >

    >
    > >

    >
    > > Concerning the rest of your problems, there is lots of code and the datafile

    >
    > > missing. However, there is also too much of it, try replacing the file with

    >
    > > generated data and remove everything from the code that is not absolutely

    >
    > > necessary.

    >
    >
    >
    > Also please copy paste the actual error message rather than paraphrasing it.
    >
    >
    >
    >
    >
    > Oscar


    I really appreciate to both Ulich and Oscar.

    To Oscar
    My actual error message is:
    File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 311, in __init__
    self._spline = splmake(x,oriented_y,order=order)
    File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 809, in splmake
    coefs = func(xk, yk, order, conds, B)
    File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 530, in _find_smoothest
    u,s,vh = np.dual.svd(B)
    File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/linalg/decomp_svd.py", line 91, in svd
    full_matrices=full_matrices, overwrite_a = overwrite_a)
    MemoryError
    --------------------------------------------------------------------------
    Thank you,

    Hoonill
    Isaac Won, Jan 23, 2013
    #8
  9. On 23 January 2013 14:28, Isaac Won <> wrote:
    > On Wednesday, January 23, 2013 4:08:13 AM UTC-6, Oscar Benjamin wrote:
    >
    > To Oscar
    > My actual error message is:
    > File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 311, in __init__
    > self._spline = splmake(x,oriented_y,order=order)
    > File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 809, in splmake
    > coefs = func(xk, yk, order, conds, B)
    > File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 530, in _find_smoothest
    > u,s,vh = np.dual.svd(B)
    > File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/linalg/decomp_svd.py", line 91, in svd
    > full_matrices=full_matrices, overwrite_a = overwrite_a)
    > MemoryError


    Are you sure that's the *whole* error message? The traceback only
    refers to the scipy modules. I can't see the line from your code that
    is generating the error.


    Oscar
    Oscar Benjamin, Jan 23, 2013
    #9
  10. Isaac Won

    Isaac Won Guest

    On Wednesday, January 23, 2013 2:55:14 AM UTC-6, Ulrich Eckhardt wrote:
    > Am 23.01.2013 05:06, schrieb Isaac Won:
    >
    > > I have tried to use different interpolation methods with Scipy. My

    >
    > > code seems just fine with linear interpolation, but shows memory

    >
    > > error with quadratic. I am a novice for python. I will appreciate any

    >
    > > help.

    >
    > >

    >
    > > #code

    >
    > > f = open(filin, "r")

    >
    >
    >
    > Check out the "with open(...) as f" syntax.
    >
    >
    >
    >
    >
    > > for columns in ( raw.strip().split() for raw in f ):

    >
    >
    >
    > For the record, this first builds a sequence and then iterates over that
    >
    > sequence. This is not very memory-efficient, try this instead:
    >
    >
    >
    > for line in f:
    >
    > columns = line.strip().split()
    >
    >
    >
    >
    >
    > Concerning the rest of your problems, there is lots of code and the
    >
    > datafile missing. However, there is also too much of it, try replacing
    >
    > the file with generated data and remove everything from the code that is
    >
    > not absolutely necessary.
    >
    >
    >
    > Good luck!
    >
    >
    >
    > Uli


    Hi Ulich,

    I tried to change the code following your advice, but it doesn't seem to work still.

    My adjusted code is:

    a = []

    with open(filin, "r") as f:

    for line in f:
    columns = line.strip().split()

    a.append(columns[5])
    x = np.array(a, float)


    not_nan = np.logical_not(np.isnan(x))
    indices = np.arange(len(x))
    interp = interp1d(indices[not_nan], x[not_nan], kind = 'quadratic')
    p = interp(indices)
    ---------------------------------------------------------------------
    And full error message is:
    interp = interp1d(indices[not_nan], x[not_nan], kind = 'quadratic')
    File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 311, in __init__
    self._spline = splmake(x,oriented_y,order=order)
    File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 809, in splmake
    coefs = func(xk, yk, order, conds, B)
    File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 530, in _find_smoothest
    u,s,vh = np.dual.svd(B)
    File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/linalg/decomp_svd.py", line 91, in svd
    full_matrices=full_matrices, overwrite_a = overwrite_a)
    MemoryError
    -----------------------------------------------------------------------
    Could you give me some advice for this situation?

    Thank you always,

    Isaac
    Isaac Won, Jan 23, 2013
    #10
  11. Isaac Won

    Isaac Won Guest

    On Wednesday, January 23, 2013 8:40:54 AM UTC-6, Oscar Benjamin wrote:
    > On 23 January 2013 14:28, Isaac Won <> wrote:
    >
    > > On Wednesday, January 23, 2013 4:08:13 AM UTC-6, Oscar Benjamin wrote:

    >
    > >

    >
    > > To Oscar

    >
    > > My actual error message is:

    >
    > > File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 311, in __init__

    >
    > > self._spline = splmake(x,oriented_y,order=order)

    >
    > > File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 809, in splmake

    >
    > > coefs = func(xk, yk, order, conds, B)

    >
    > > File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 530, in _find_smoothest

    >
    > > u,s,vh = np.dual.svd(B)

    >
    > > File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/linalg/decomp_svd.py", line 91, in svd

    >
    > > full_matrices=full_matrices, overwrite_a = overwrite_a)

    >
    > > MemoryError

    >
    >
    >
    > Are you sure that's the *whole* error message? The traceback only
    >
    > refers to the scipy modules. I can't see the line from your code that
    >
    > is generating the error.
    >
    >
    >
    >
    >
    > Oscar


    Dear Oscar,

    Following is full error message after I adjusted following Ulich's advice:

    interp = interp1d(indices[not_nan], x[not_nan], kind = 'quadratic')
    File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 311, in __init__
    self._spline = splmake(x,oriented_y,order=order)
    File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 809, in splmake
    coefs = func(xk, yk, order, conds, B)
    File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 530, in _find_smoothest
    u,s,vh = np.dual.svd(B)
    File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/linalg/decomp_svd.py", line 91, in svd
    full_matrices=full_matrices, overwrite_a = overwrite_a)
    MemoryError
    ----------------------------------------------------------------------
    Thank you,

    Hoonill
    Isaac Won, Jan 23, 2013
    #11
  12. Isaac Won

    Isaac Won Guest

    On Wednesday, January 23, 2013 8:40:54 AM UTC-6, Oscar Benjamin wrote:
    > On 23 January 2013 14:28, Isaac Won <> wrote:
    >
    > > On Wednesday, January 23, 2013 4:08:13 AM UTC-6, Oscar Benjamin wrote:

    >
    > >

    >
    > > To Oscar

    >
    > > My actual error message is:

    >
    > > File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 311, in __init__

    >
    > > self._spline = splmake(x,oriented_y,order=order)

    >
    > > File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 809, in splmake

    >
    > > coefs = func(xk, yk, order, conds, B)

    >
    > > File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 530, in _find_smoothest

    >
    > > u,s,vh = np.dual.svd(B)

    >
    > > File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/linalg/decomp_svd.py", line 91, in svd

    >
    > > full_matrices=full_matrices, overwrite_a = overwrite_a)

    >
    > > MemoryError

    >
    >
    >
    > Are you sure that's the *whole* error message? The traceback only
    >
    > refers to the scipy modules. I can't see the line from your code that
    >
    > is generating the error.
    >
    >
    >
    >
    >
    > Oscar


    Dear Oscar,

    Following is full error message after I adjusted following Ulich's advice:

    interp = interp1d(indices[not_nan], x[not_nan], kind = 'quadratic')
    File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 311, in __init__
    self._spline = splmake(x,oriented_y,order=order)
    File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 809, in splmake
    coefs = func(xk, yk, order, conds, B)
    File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 530, in _find_smoothest
    u,s,vh = np.dual.svd(B)
    File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/linalg/decomp_svd.py", line 91, in svd
    full_matrices=full_matrices, overwrite_a = overwrite_a)
    MemoryError
    ----------------------------------------------------------------------
    Thank you,

    Hoonill
    Isaac Won, Jan 23, 2013
    #12
  13. On 23 January 2013 14:57, Isaac Won <> wrote:
    > On Wednesday, January 23, 2013 8:40:54 AM UTC-6, Oscar Benjamin wrote:
    >> On 23 January 2013 14:28, Isaac Won <> wrote:
    >>

    [SNIP]
    >
    > Following is full error message after I adjusted following Ulich's advice:
    >
    > interp = interp1d(indices[not_nan], x[not_nan], kind = 'quadratic')
    > File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 311, in __init__
    > self._spline = splmake(x,oriented_y,order=order)
    > File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 809, in splmake
    > coefs = func(xk, yk, order, conds, B)
    > File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 530, in _find_smoothest
    > u,s,vh = np.dual.svd(B)
    > File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/linalg/decomp_svd.py", line 91, in svd
    > full_matrices=full_matrices, overwrite_a = overwrite_a)
    > MemoryError


    Where is the new code? You should show full working code (with the
    import statements) and the full error that is generated by exactly
    that code. If possible you should also write code that someone else
    could run even without having access to your data files. If you did
    that in your first post, you'd probably have an answer to your problem
    by now.

    Here is a version of your code that many people on this list can test
    straight away:

    import numpy as np
    from scipy.interpolate import interp1d
    x = np.array(31747 * [0.0], float)
    indices = np.arange(len(x))
    interp = interp1d(indices, x, kind='quadratic')

    Running this gives the following error:

    ~$ python tmp.py
    Traceback (most recent call last):
    File "tmp.py", line 5, in <module>
    interp = interp1d(indices, x, kind='quadratic')
    File "/usr/lib/python2.7/dist-packages/scipy/interpolate/interpolate.py",
    line 308, in __init__
    self._spline = splmake(x,oriented_y,order=order)
    File "/usr/lib/python2.7/dist-packages/scipy/interpolate/interpolate.py",
    line 805, in splmake
    B = _fitpack._bsplmat(order, xk)
    MemoryError

    Unless I've misunderstood how this function is supposed to be used, it
    just doesn't really seem to work for arrays of much more than a few
    hundred elements.


    Oscar
    Oscar Benjamin, Jan 23, 2013
    #13
  14. Isaac Won

    Isaac Won Guest

    On Wednesday, January 23, 2013 10:51:43 AM UTC-6, Oscar Benjamin wrote:
    > On 23 January 2013 14:57, Isaac Won <> wrote:
    >
    > > On Wednesday, January 23, 2013 8:40:54 AM UTC-6, Oscar Benjamin wrote:

    >
    > >> On 23 January 2013 14:28, Isaac Won <> wrote:

    >
    > >>

    >
    > [SNIP]
    >
    > >

    >
    > > Following is full error message after I adjusted following Ulich's advice:

    >
    > >

    >
    > > interp = interp1d(indices[not_nan], x[not_nan], kind = 'quadratic')

    >
    > > File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 311, in __init__

    >
    > > self._spline = splmake(x,oriented_y,order=order)

    >
    > > File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 809, in splmake

    >
    > > coefs = func(xk, yk, order, conds, B)

    >
    > > File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 530, in _find_smoothest

    >
    > > u,s,vh = np.dual.svd(B)

    >
    > > File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/linalg/decomp_svd.py", line 91, in svd

    >
    > > full_matrices=full_matrices, overwrite_a = overwrite_a)

    >
    > > MemoryError

    >
    >
    >
    > Where is the new code? You should show full working code (with the
    >
    > import statements) and the full error that is generated by exactly
    >
    > that code. If possible you should also write code that someone else
    >
    > could run even without having access to your data files. If you did
    >
    > that in your first post, you'd probably have an answer to your problem
    >
    > by now.
    >
    >
    >
    > Here is a version of your code that many people on this list can test
    >
    > straight away:
    >
    >
    >
    > import numpy as np
    >
    > from scipy.interpolate import interp1d
    >
    > x = np.array(31747 * [0.0], float)
    >
    > indices = np.arange(len(x))
    >
    > interp = interp1d(indices, x, kind='quadratic')
    >
    >
    >
    > Running this gives the following error:
    >
    >
    >
    > ~$ python tmp.py
    >
    > Traceback (most recent call last):
    >
    > File "tmp.py", line 5, in <module>
    >
    > interp = interp1d(indices, x, kind='quadratic')
    >
    > File "/usr/lib/python2.7/dist-packages/scipy/interpolate/interpolate.py",
    >
    > line 308, in __init__
    >
    > self._spline = splmake(x,oriented_y,order=order)
    >
    > File "/usr/lib/python2.7/dist-packages/scipy/interpolate/interpolate.py",
    >
    > line 805, in splmake
    >
    > B = _fitpack._bsplmat(order, xk)
    >
    > MemoryError
    >
    >
    >
    > Unless I've misunderstood how this function is supposed to be used, it
    >
    > just doesn't really seem to work for arrays of much more than a few
    >
    > hundred elements.
    >
    >
    >
    >
    >
    > Oscar


    Thank you Oscar for your help and advice.

    I agree with you. So, I tried to find the way to solve this problem.

    My full code adjusted is:
    from scipy.interpolate import interp1d

    import numpy as np
    import matplotlib.pyplot as plt



    with open(filin, "r") as f:

    for line in f:
    columns = line.strip().split()

    a.append(columns[5])
    x = np.array(a, float)


    not_nan = np.logical_not(np.isnan(x))
    indices = np.arange(len(x))
    interp = interp1d(indices[not_nan], x[not_nan], kind = 'quadratic')

    p = interp(indices)




    k = np.arange(31747)

    plt.subplot(211)
    plt.plot(k, p)
    plt.xlabel('Quadratic interpolation')
    plt.subplot(212)
    plt.plot(k, x)

    plt.show()
    -----------------------------------------------------------------
    Whole error message was:

    Traceback (most recent call last):
    File "QI1.py", line 22, in <module>
    interp = interp1d(indices[not_nan], x[not_nan], kind = 'quadratic')
    File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 311, in __init__
    self._spline = splmake(x,oriented_y,order=order)
    File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 809, in splmake
    coefs = func(xk, yk, order, conds, B)
    File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 530, in _find_smoothest
    u,s,vh = np.dual.svd(B)
    File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/linalg/decomp_svd.py", line 91, in svd
    full_matrices=full_matrices, overwrite_a = overwrite_a)
    MemoryError
    ----------------------------------------------------------------------
    Thank you again Oscar,

    Isaac
    Isaac Won, Jan 23, 2013
    #14
  15. Isaac Won

    Isaac Won Guest

    On Wednesday, January 23, 2013 10:51:43 AM UTC-6, Oscar Benjamin wrote:
    > On 23 January 2013 14:57, Isaac Won <> wrote:
    >
    > > On Wednesday, January 23, 2013 8:40:54 AM UTC-6, Oscar Benjamin wrote:

    >
    > >> On 23 January 2013 14:28, Isaac Won <> wrote:

    >
    > >>

    >
    > [SNIP]
    >
    > >

    >
    > > Following is full error message after I adjusted following Ulich's advice:

    >
    > >

    >
    > > interp = interp1d(indices[not_nan], x[not_nan], kind = 'quadratic')

    >
    > > File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 311, in __init__

    >
    > > self._spline = splmake(x,oriented_y,order=order)

    >
    > > File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 809, in splmake

    >
    > > coefs = func(xk, yk, order, conds, B)

    >
    > > File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 530, in _find_smoothest

    >
    > > u,s,vh = np.dual.svd(B)

    >
    > > File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/linalg/decomp_svd.py", line 91, in svd

    >
    > > full_matrices=full_matrices, overwrite_a = overwrite_a)

    >
    > > MemoryError

    >
    >
    >
    > Where is the new code? You should show full working code (with the
    >
    > import statements) and the full error that is generated by exactly
    >
    > that code. If possible you should also write code that someone else
    >
    > could run even without having access to your data files. If you did
    >
    > that in your first post, you'd probably have an answer to your problem
    >
    > by now.
    >
    >
    >
    > Here is a version of your code that many people on this list can test
    >
    > straight away:
    >
    >
    >
    > import numpy as np
    >
    > from scipy.interpolate import interp1d
    >
    > x = np.array(31747 * [0.0], float)
    >
    > indices = np.arange(len(x))
    >
    > interp = interp1d(indices, x, kind='quadratic')
    >
    >
    >
    > Running this gives the following error:
    >
    >
    >
    > ~$ python tmp.py
    >
    > Traceback (most recent call last):
    >
    > File "tmp.py", line 5, in <module>
    >
    > interp = interp1d(indices, x, kind='quadratic')
    >
    > File "/usr/lib/python2.7/dist-packages/scipy/interpolate/interpolate.py",
    >
    > line 308, in __init__
    >
    > self._spline = splmake(x,oriented_y,order=order)
    >
    > File "/usr/lib/python2.7/dist-packages/scipy/interpolate/interpolate.py",
    >
    > line 805, in splmake
    >
    > B = _fitpack._bsplmat(order, xk)
    >
    > MemoryError
    >
    >
    >
    > Unless I've misunderstood how this function is supposed to be used, it
    >
    > just doesn't really seem to work for arrays of much more than a few
    >
    > hundred elements.
    >
    >
    >
    >
    >
    > Oscar


    Thank you Oscar for your help and advice.

    I agree with you. So, I tried to find the way to solve this problem.

    My full code adjusted is:
    from scipy.interpolate import interp1d

    import numpy as np
    import matplotlib.pyplot as plt



    with open(filin, "r") as f:

    for line in f:
    columns = line.strip().split()

    a.append(columns[5])
    x = np.array(a, float)


    not_nan = np.logical_not(np.isnan(x))
    indices = np.arange(len(x))
    interp = interp1d(indices[not_nan], x[not_nan], kind = 'quadratic')

    p = interp(indices)




    k = np.arange(31747)

    plt.subplot(211)
    plt.plot(k, p)
    plt.xlabel('Quadratic interpolation')
    plt.subplot(212)
    plt.plot(k, x)

    plt.show()
    -----------------------------------------------------------------
    Whole error message was:

    Traceback (most recent call last):
    File "QI1.py", line 22, in <module>
    interp = interp1d(indices[not_nan], x[not_nan], kind = 'quadratic')
    File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 311, in __init__
    self._spline = splmake(x,oriented_y,order=order)
    File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 809, in splmake
    coefs = func(xk, yk, order, conds, B)
    File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/interpolate/interpolate.py", line 530, in _find_smoothest
    u,s,vh = np.dual.svd(B)
    File "/lustre/work/apps/python-2.7.1/lib/python2.7/site-packages/scipy/linalg/decomp_svd.py", line 91, in svd
    full_matrices=full_matrices, overwrite_a = overwrite_a)
    MemoryError
    ----------------------------------------------------------------------
    Thank you again Oscar,

    Isaac
    Isaac Won, Jan 23, 2013
    #15
  16. On 23 January 2013 17:33, Isaac Won <> wrote:
    > On Wednesday, January 23, 2013 10:51:43 AM UTC-6, Oscar Benjamin wrote:
    >> On 23 January 2013 14:57, Isaac Won <> wrote:
    >>
    >> > On Wednesday, January 23, 2013 8:40:54 AM UTC-6, Oscar Benjamin wrote:

    >>
    >> Unless I've misunderstood how this function is supposed to be used, it
    >> just doesn't really seem to work for arrays of much more than a few
    >> hundred elements.
    >>


    The solution is to use UnivariateSpline. I don't know what the
    difference is but it works where the other fails:

    import numpy as np
    from scipy.interpolate import UnivariateSpline
    x = np.array(10000 * [0.0], float)
    indices = np.arange(len(x))
    interp = UnivariateSpline(indices, x, k=2)
    print(interp(1.5))


    Oscar
    Oscar Benjamin, Jan 24, 2013
    #16
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Vinodh Kumar

    Quadratic curve fitting!

    Vinodh Kumar, Aug 11, 2003, in forum: C++
    Replies:
    6
    Views:
    5,200
    onetimeuser
    Dec 1, 2010
  2. Philip Smith

    Multiple Polynomial Quadratic Sieve

    Philip Smith, May 30, 2006, in forum: Python
    Replies:
    2
    Views:
    518
    Philip Smith
    May 30, 2006
  3. Replies:
    3
    Views:
    430
  4. fb

    Quadratic Formula Woes

    fb, Aug 30, 2004, in forum: C Programming
    Replies:
    6
    Views:
    342
    Karthik Kumar
    Sep 1, 2004
  5. Totti
    Replies:
    5
    Views:
    1,411
    Totti
    Dec 22, 2007
Loading...

Share This Page