Bug/Gross InEfficiency in HeathField's fgetline program

[Tor Rustad]

Even for time value of money calculations (e.g. compound interest)?

I haven't programmed or audited code for compound interest calculations,
but when I analyzed all the COBOL sources (for Y2K bugs) of a major
in-house system, no floating-point code was scanned (approx 300.000 LOC).

COBOL has built in support for fixed-point calculations, so when dealing
with monetary computations on mainframes, that's the natural pick IMO.

 

[user923005]

I haven't programmed or audited code for compound interest calculations,
but when I analyzed all the COBOL sources (for Y2K bugs) of a major
in-house system, no floating-point code was scanned (approx 300.000 LOC).

COBOL has built in support for fixed-point calculations, so when dealing
with monetary computations on mainframes, that's the natural pick IMO.

I would be very leery of fixed point calculations used to calculate
interest over time.

I use a modified version of Moshier's Qfloat and perform all
calculations with 100 digits of precision. It can calculate "U.S.
national debt" sized amounts over centuries and not lose a penny.

Of course, it is possible that your COBOL compiler has done something
like that.

 

[Tor Rustad]

Tor Rustad wrote, On 29/10/07 23:50:
[...]

Performance requirements might be important, but correctness, risk,
fault-tolerance, disaster-recovery, scalability, fail-over-time etc.
are typically more important, at least if Gordon is not talking about
some back-office system, but rather an online transaction environment,
like e.g. a stock-exchange or an EFTPOS host.

It is a major back-office system. It is the system that a number of
large companies are using for entering all their requisitions, orders,
invoices and goods received notes, matching invoices to GRNs and lots of
other stuff. Also lots of reporting on all this data. It is business
critical for the companies using it.

OK, if availability requirements (SLA) of this back-office system, are
near a major OLTP host, my point should still hold though.

Actually, the first rule is do not even consider looking at it or
bothering doing any measurements unless there is known to be a good
reason to start looking. As long as the code is fast enough (which it is
currently) we will concentrate on writing code to meet new and changed
requirements as correctly as possible and fixing the existing bugs. We
will only worry about measuring when there is a performance problem
significant enough to be worth the costs of investigating it.

Unless the transaction volume (and system) is rather static,
measurements are useful for capacity planning too. Since, the system is
business-critical for multiple companies, doing some measurements is
needed, view it as a stress test in the production environment.
Bottlenecks, can be an early-warning of a bug, or mis-configuration.

 

[Flash Gordon]

Tor Rustad wrote, On 31/10/07 00:14:

Tor Rustad wrote, On 29/10/07 23:50:
[...]

Performance requirements might be important, but correctness, risk,
fault-tolerance, disaster-recovery, scalability, fail-over-time etc.
are typically more important, at least if Gordon is not talking about
some back-office system, but rather an online transaction
environment, like e.g. a stock-exchange or an EFTPOS host.

It is a major back-office system. It is the system that a number of
large companies are using for entering all their requisitions, orders,
invoices and goods received notes, matching invoices to GRNs and lots
of other stuff. Also lots of reporting on all this data. It is
business critical for the companies using it.

OK, if availability requirements (SLA) of this back-office system, are
near a major OLTP host, my point should still hold though.

Well, a hundred people sitting twiddling their thumbs is expensive, as
is being fined for not getting your monthly returns in to the government
on time

Unless the transaction volume (and system) is rather static,

For any given customer it tends to be fairly static once they are up and
running.

measurements are useful for capacity planning too.

Well, we have measurements of how it performs in the real world, i.e.
how many users customers manage support on a given size of server with
an acceptable level of performance.

Since, the system is
business-critical for multiple companies, doing some measurements is
needed, view it as a stress test in the production environment.

Ah, but we have the *real* stress test of real stressed users ;-)

Bottlenecks, can be an early-warning of a bug, or mis-configuration.

Well, each customer has there system configured differently because they
all use it differently. Note that this is SW we sell to customers that
they run on their own servers.

Seriously, we (the company) have a lot of experience of how the system
performs in the real world, so those doing the testing of new releases
notice if the performance has been reduced significantly. Also because
of all this experience we don't need further measurements to tell
customers what size of server they need. With certain major upgrades
were we know performance could be affected we do some performance
testing, and if we ever need to do real optimisation we will do some
profiling (I've even found a suitable tool for us to use). However,
until we need to do optimisation we have more important things to do,
things which our customers consider more important.

 

[Malcolm McLean]

No, most financial institutions are not even allowed to use floating-point
for financial calculations. In fact, I have only seen fixed point
calculations, when dealing with money.

The mean of a set of integers is rational. It is not necessarily an integer.

 

[Richard Bos]

The mean of a set of integers is rational. It is not necessarily an integer.

That would argue for using a rational type, not a floating type.

Richard

 

[Tor Rustad]

The mean of a set of integers is rational. It is not necessarily an
integer.

So, the mean can (usually) be represented internally exact.

Furthermore, many of the calculations needed for double entry
bookeeping, can be done exactly by banks.

 

[Richard Heathfield]

Tor Rustad said:

So, the mean can (usually) be represented internally exact.

Furthermore, many of the calculations needed for double entry
bookeeping, can be done exactly by banks.

Why "many" rather than "all"? Since double-entry book-keeping is a way of
recording financial transactions (i.e. transfers of an exact integer
number of base currency units, e.g. pennies, cents, pesos or whatever),
I'm struggling to imagine any real-world double-entry-relevant calculation
that /cannot/ be done exactly. Even if the numbers involved are colossal,
they can still be processed exactly. You might need a bignum lib to do it,
but you *can* do it.

 

[Tor Rustad]

Tor Rustad said:


Why "many" rather than "all"? Since double-entry book-keeping is a way of
recording financial transactions (i.e. transfers of an exact integer
number of base currency units, e.g. pennies, cents, pesos or whatever),
I'm struggling to imagine any real-world double-entry-relevant calculation
that /cannot/ be done exactly. Even if the numbers involved are colossal,
they can still be processed exactly. You might need a bignum lib to do it,
but you *can* do it.

What if the currency of the transaction, is different from the currency
of the account?

We still have only one currency here, but e.g. UK banks could perhaps
accept transactions in Euro's too? How was the transition to Euro done
in France, Germany etc?

 

[Richard Heathfield]

Tor Rustad said:

What if the currency of the transaction, is different from the currency
of the account?

I think you'll find that the banks fudge the amount so that it comes to a
whole number of pennies/cents/whatever.

We still have only one currency here, but e.g. UK banks could perhaps
accept transactions in Euro's too? How was the transition to Euro done
in France, Germany etc?

Badly, it seems. In 1997, 1 Euro was supposed to be worth around 73p, and
it's currently only worth about 69p, so it's dropped by about 5.5% over 10
years.

 

[Francine.Neary]

Badly, it seems. In 1997, 1 Euro was supposed to be worth around 73p, and
it's currently only worth about 69p, so it's dropped by about 5.5% over 10
years.

More accurately, both currencies have declined in value (due to
inflation) but the Euro has depreciated more rapidly than the Pound
since 1997.

 

[Tor Rustad]

Tor Rustad said:


I think you'll find that the banks fudge the amount so that it comes to a
whole number of pennies/cents/whatever.

I don't know what "fudge" means, but I assume you still argue your
point, as you always do!

I expect, account transfers in e.g. UK these days, to be available
between sterling and euro accounts, and such conversion could result in
rounding. That's the reason I wrote *many*, and not *all*.

I'm right, or I'm right?

 

[Richard Heathfield]

Tor Rustad said:

I don't know what "fudge" means,

fiddle, tweak, finagle, frob...

but I assume you still argue your
point, as you always do!

Well, it's funny you should say that, because I'm going to stop arguing it
here. "I think you'll find" means precisely that - i.e. what I've said is
what I think happens, but I could certainly be wrong.

I expect, account transfers in e.g. UK these days, to be available
between sterling and euro accounts, and such conversion could result in
rounding. That's the reason I wrote *many*, and not *all*.

I'm right, or I'm right?

I don't know whether or not you are right. I suspect that you're mistaken,
but I could easily be wrong.

 

[Joe Wright]

More accurately, both currencies have declined in value (due to
inflation) but the Euro has depreciated more rapidly than the Pound
since 1997.

The Euro and the Pound have declined against what? The Pound buys
US$2.08 now. The Euro buys $1.45 or so. They seem strong against the dollar.

 

[Malcolm McLean]

Tor Rustad said:


I think you'll find that the banks fudge the amount so that it comes to a
whole number of pennies/cents/whatever.

Badly, it seems. In 1997, 1 Euro was supposed to be worth around 73p,
and it's currently only worth about 69p, so it's dropped by about 5.5%
over
10 years.

1 Euro isn't worth 69p. If you can trade it at all you'd probably get more
like 50p for it. Trade thousands of Euros and you'll be able to get better
terms.

That's the answer to Tor's question. A bank cannot covert 100 Euros to 69
pounds 21.23456p applying the exchange rate. It's got to trade those Euros
with someone who has pounds. And it will settle on a whole number of
pennies.

 

[Flash Gordon]

Malcolm McLean wrote, On 03/11/07 07:13:

1 Euro isn't worth 69p. If you can trade it at all you'd probably get
more like 50p for it. Trade thousands of Euros and you'll be able to get
better terms.

That's the answer to Tor's question. A bank cannot covert 100 Euros to
69 pounds 21.23456p applying the exchange rate. It's got to trade those
Euros with someone who has pounds. And it will settle on a whole number
of pennies.

Similarly with all other transactions and accounting practices that I am
aware of. You are always *required* to round (either up to, down to or
to the nearest) whole whatever. The exact rounding rules change
depending on what you are doing, but they are always exactly specified.

Oh, and in our multi-currency system we store the amount in the original
currency and the exchange rate on each transaction so we can do exact
calculation in originating currency and correctly rounded calculations
in the target currency. This adds a whole load more integer calculations
on money to the SW.

 

[Tor Rustad]

Tor Rustad said:


fiddle, tweak, finagle, frob...

So, you talked about *rounding*.

FYI, calculations using rounding, are not done *exactly*.

 

[Francine.Neary]

The Euro and the Pound have declined against what? The Pound buys
US$2.08 now. The Euro buys $1.45 or so. They seem strong against the dollar.

They have declined absolutely. Money is worth what you can buy with
it, and a Pound today buys you less stuff than a Pound did in 1997.
(Similarly with a Euro, had there been any actual Euros in 1997.) Your
observation is that the Dollar has declined more rapidly than the
Pound and the Euro in that time.

 

[Malcolm McLean]

Malcolm McLean wrote, On 03/11/07 07:13:

Similarly with all other transactions and accounting practices that I am
aware of. You are always *required* to round (either up to, down to or to
the nearest) whole whatever. The exact rounding rules change depending on
what you are doing, but they are always exactly specified.

Oh, and in our multi-currency system we store the amount in the original
currency and the exchange rate on each transaction so we can do exact
calculation in originating currency and correctly rounded calculations in
the target currency. This adds a whole load more integer calculations on
money to the SW.

You can use a double to hold integers. It has got several advantages, such
as being able to hold more digits of precision than a 32 bit type, degrading
gracefully under hyperinflation - it will no longer give exact results, of
course, but if there are 6 trillion dollars to the pound that will be the
least of any American's worries - and giving you a free check that rounding
has actually been applied.

 

[Keith Thompson]

You can use a double to hold integers. It has got several advantages,
such as being able to hold more digits of precision than a 32 bit
type, degrading gracefully under hyperinflation - it will no longer
give exact results, of course, but if there are 6 trillion dollars to
the pound that will be the least of any American's worries - and
giving you a free check that rounding has actually been applied.

As I understand it, inexact results in monetary calculations are
*never* acceptable. Using a floating-point type might give you an
acceptable range over which exact values can be represented, but if
you go outside that range you *silently* get inexact results.

In a typical modern implementation, both double and long long are 64
bits -- but long long doesn't waste bits on an exponent, and can
therefore represent a much wider range of integer values exactly.
Many implementations that don't support long long nevertheless support
a 64-bit integer type by another name.