steepest slope estimate

[jackson marshmallow]

Hi everyone,

I need to write a program to estimate the steepest slope in a sequence of
noisy data points. One approach would be to make a series of linear fits and
vary the window size; choose the line with the steepest slope for each
window size; choose the line that fits best, according to some criterion.

Am I reinventing the wheel? Can you suggest any existing algorithms, books,
web pages?

Thanks in advance for your help.

 

[Roedy Green]

I need to write a program to estimate the steepest slope in a sequence of
noisy data points. One approach would be to make a series of linear fits and
vary the window size; choose the line with the steepest slope for each
window size; choose the line that fits best, according to some criterion.

If you literally want the steepest slope, you just compute the slope
between pairs and keep the biggest. However, I suspect that is not
what you want. You want to smooth the curve first THEN take the
steepest slope. If you fit to some curve, you can algebraically take
the derivative of that, and use Newton Raphson type methods to home in
on local maxima.

 

[jackson marshmallow]

If you literally want the steepest slope, you just compute the slope
between pairs and keep the biggest. However, I suspect that is not
what you want. You want to smooth the curve first THEN take the
steepest slope. If you fit to some curve, you can algebraically take
the derivative of that, and use Newton Raphson type methods to home in
on local maxima.

Thanks for you reply!

The problem is in smoothing the curve. I cannot make any assumptions about
the window size...

 

[Roedy Green]

The problem is in smoothing the curve. I cannot make any assumptions about
the window size...

Some simple smoothing would be to do a moving average where each point
is a weighted average of nearby points. Just fidddle with the weights
till you get something that looks smooth enough.

If you want something that is mathematically meaningful, you will have
to study a statistics text.

 

[nos]

polynomial fit is popular you should be able to find
code somewhere

 

[Roedy Green]

polynomial fit is popular you should be able to find
code somewhere

hermite, Chebychev ...

 

[Guest]

Thanks for you reply!

The problem is in smoothing the curve. I cannot make any assumptions about
the window size...

Really? This seems odd. I solved a problem like this many, many years
ago (on an Apple II used for automated data collection) by the "smooth
then search" method. But of course I did know what the window should
be, based on the response characteristics of the spectrometer that was
generating the data.

If you have real data from a real source you must have some idea of
a window size. Or are you trying to solve a theoretical problem
here? What if the window size is less that the resolution between
two data points, or larger than the entire set? Nonsense in both
cases. This is why I'm wondering if you really have no idea about the
window size.

--arne

DISCLAIMER: These opinions and statements are those of the author and
do not represent any views or positions of the Hewlett-Packard Co.

 

[jackson marshmallow]

Really? This seems odd. I solved a problem like this many, many years
ago (on an Apple II used for automated data collection) by the "smooth
then search" method. But of course I did know what the window should
be, based on the response characteristics of the spectrometer that was
generating the data.

If you have real data from a real source you must have some idea of
a window size. Or are you trying to solve a theoretical problem
here? What if the window size is less that the resolution between
two data points, or larger than the entire set? Nonsense in both
cases. This is why I'm wondering if you really have no idea about the
window size.

Thanks for your reply.

No, this is a practical problem of finding the transconductance of FETs from
automatically collected data.

Of course there are limits of the window size, but I can't just pick a
number of points for a linear fit that will work for all devices.

 

[Guest]

Thanks for your reply.

No, this is a practical problem of finding the transconductance of FETs from
automatically collected data.

Of course there are limits of the window size, but I can't just pick a
number of points for a linear fit that will work for all devices.

Well then, all I can say is that this sounds like what my old
DiffEQ prof would have called an "interesting problem"... ;-)

It's beyond my knowledge how to solve it. You might do better
looking in some advanced numerical analysis books then asking
here though. No disrespect to this group, but I think you got
a math problem here, not a Java problem.

--arne

 

[nos]

I still think polynomial fit is best
1. easy to find code on web to do the fit
2. easy to find slope analytically and code for that

 

[Guest]

I still think polynomial fit is best
1. easy to find code on web to do the fit
2. easy to find slope analytically and code for that

3. doesn't usually work for any algorithms that
depend on finding slope

Been there, done that, failed.

Generally you need to know the analytic form you
are trying to fit to. Polynomial curve fitting fits
data that happens to be produced by a phenomenon
that is in fact modeled by an order-n polynomial,
and BTW you have to know the order. Otherwise
all you are drawing is a pretty curved line that
is useless.

If you want to convince yourself of this do the
following:

1. Generate a data set from some non-polynomial
formula that still looks enough like a polynomial
to be fit (say the center part of a normal
distribution curve).

2. Add noise.

3. Apply a polynomial fit to the resulting data.
Pick whatever order suits your fancy.

4. Take the analytic derivative of the resulting
function and compare it to the analytic derivative
of the original function.

It's unlikely they will even be close. Trust me,
I spent a fair chunk of my time in grad school
on problems like this. It isn't that easy.

--arne

DISCLAIMER: These opinions and statements are those of the author and
do not represent any views or positions of the Hewlett-Packard Co.

 

[nos]

tnx for the info, very interesting stuff
i usually just use order 12 or so as that
seems to fit almost anything, but i never
looked at the slope as you suggest

 

[jackson marshmallow]

3. doesn't usually work for any algorithms that
depend on finding slope

Been there, done that, failed.

Generally you need to know the analytic form you
are trying to fit to. Polynomial curve fitting fits
data that happens to be produced by a phenomenon
that is in fact modeled by an order-n polynomial,
and BTW you have to know the order. Otherwise
all you are drawing is a pretty curved line that
is useless.

You're right, in my case, polynominal fit would be nothing than a pretty
curved line.

The actual curve is expected to have 4 regions:

I flat
II some monotonic function (negative derivative)
III linear decline (this is the steepest slope I'm looking for)
IV exponential decay

Thanks a lot for your comments!

I actually posted this first on sci.math.num-analysis...