Linux's approaching Achilles' heal

[Richard Heathfield]

CBFalconer said:

Barry Schwarz wrote:


In addition, malloc and friends have no idea what earlier
assignments are being used for. There is no reason the program
cannot react to the error by releasing stored items until this
malloc actually succeeds. As long as success can be thus attained,
there is no reason for the program to fail.

Unless of course some brain-damaged library designer has arranged that
malloc never, ever returns NULL, so that even properly-written
applications can't tell whether a request succeeded or not. If ever there
was a decision that favoured the mediocre over the competent, that's the
one.

 

[Robert Redelmeier]

The main problem with your argument being, of course, that Vista
which was delayed several times and had features thrown out so that
it could finally come to market, seems to have even more problems.

This is a valid high-level argument: success is much more than
avoiding failure. Even glaring failures can be immaterial.

However, I am reading in ALA where details are very relevant so
I feel compelled to offer some of the many:


1) A Linux distro is _not_ the kernel. distros come and go,
the kernel is eternal

2) Much greater code review has been done for OpenBSD. It has
not lead to run-away success outside of its' domain.

3) Using a desktop/user distro for a "critical support system"
is unlikely to be successful except for "non-traditional"
definitions of "critical"

4) audio might be one of those defininitions

5) not understanding VM_overcommit and the OOM killer certainly
is "non-traditional" wrt "critical".

6) If Nathan or J-G dislike certainly library coding, they are
completely free to change it, forking the project. This is one
of the great strengths of the GPL and Linux. Whining is very
poor form and a waste of effort. Projects propagate based on
user judgement. Not critics and whiners.


-- Robert

 

[Joe]

Sigh, had to go to Google to read the other six posts that didn't propagate
well...


Although I strongly believe there are reasons to support the claim that
Linux is or will be "tumbling down the hill into the valley of the
forgotten, non-important OSs that 'could have been'," I don't believe the
issue is the mindset of Linux coders, their standards, their failure to fix
bugs, or even other issues such as reversion of prior bug fixes or
filesystem problems...

The real primary issue is money. Can Linux survive long term against a
company with billions in financial and physical capital, licensed and
proprietary software patents, driven programmers who are _paid_ to program
for a living, and an endless supply of software drivers written for their
OS's API by hardware manufacturers. Secondary issues include software
development time for new PC hardware or circuitry and the far above average
intellect of "their" large paid programmer base versus the average IQ,
skill, and time constraints of many unpaid "Joe Six-pack" 's. I see Linux
running into a wall due to the rapid continuous changes and advances in PC
circuitry unless a huge infusion of cash is found.

This has been the situation for the last twenty years. Linux and GNU
were born into and grew up in exactly this environment. If they die now,
it won't be for this reason.

Microsoft certainly has good people working for it. But they are very
closely constrained by the requirement to keep re-selling what is
broadly the same software, and even more so by the importance of
maintaining the near-monopoly. What innovation does occur is almost
entirely aimed at keeping and improving the incompatibility between
Windows and the rest of the IT world, and to some extent even with
earlier Microsoft software. GNU-Linux has no need or use for planned
obsolescence.

One particularly crippling constraint is that much-loved marketing word
'integration'. This means linking together relatively unrelated programs
so tightly that connection with non-Microsoft software is difficult or
impossible. This is the exact opposite of what is probably the single
strongest programming imperative, to isolate sub-programs as much as
possible and to use only well-defined interfaces between them.

A simple example: the Windows Small Business Server contains a POP3
downloader which drops mail straight into Exchange mailboxes, because it
can, and because the suits can then use the 'i' word. The competitive
POP3 download products all deliver to localhost:25 by SMTP, keeping the
interface clean and simple. The result is that the competitors can
utilise a number of Exchange features which the built-in POP3 connector
bypasses. While Microsoft is not a company to be underestimated, it
should not be overestimated either.

 

[Barry Schwarz]

That's odd... I was under the impression we had this thing called
paging, on modern operating systems. This has two effects; one,
applications are actually allocated memory in complete pages, and
secondly, those pages can reside anywhere in physical ram, and they'll
still appear contiguous to the application.

It is true that a contiguous block of virtual memory need not be
located in a contiguous block of real memory. It is also irrelevant
to the point being made.

Once a virtual address has been allocated it cannot be reused until
it is freed. If multiple allocation requests for 65K blocks have
returned virtual addresses of 0x10000, 0x20000, and 0x30000 and I free
the middle one, a request for 130K will not fit at 0x20000. But a
request for 10K will. Virtual memory is subject to the same
fragmentation problems as real memory.

The only time this might be an issue, is with DMA, where a component
external to the processor (and hence without the benefit of the kernels
page tables) needs to access data across two or more pages.

The real memory address are not the issue.

Mind you, I'm not a systems level programmer either...



You, mate, are an ass. Every time I have run out of memory on a
Windoze system, the entire system crashed. My wife who still uses

Before, you were merely technically incorrect. Now you are no longer
worth reading.

<snip>


Remove del for email

 

[pete]

It is true that a contiguous block of virtual memory need not be
located in a contiguous block of real memory. It is also irrelevant
to the point being made.

It's completely irrelevant to the point being made.

 

[Jerry McBride]

Like a run-away freighttrain, the Open Source Community's "standard
practice" (_faux peer review_ plus shoddy coding standards and casual
dismissal of bug reports pointing out critical flaws
http://pulseaudio.org/ticket/158 ) is exactly the mind-set that will bring
Linux tumbling down the hill into the valley of the forgotten,
non-important OSs that "could have been".

It is easy to understand that, given the pressure to maintain a
'presence' in the month headlines and the desire to outperform the
competition in the number of 'features', some amount of short-cuts
will be taken and code audits being skipped so that the next 'distro
release' can announce a new fancy gizmo under its wing. *Some* degree
of this behavior is to be expected in an environment where any "Joe
Six-pack" can start a project and have his code used by and
encorporated into other software down the stream. However, I am quite
shocked that the practice is tolerated to the point that it leads to
extremely unstable critical support systems as detailed in the
following forum threads.

http://ubuntuforums.org/showthread.php?t=612606
http://ubuntuforums.org/showthread.php?t=614962

Nathan.

FUD... FUD... Go away... Come again... Some other day.

If it smells like a troll, looks like a troll and writes like a troll...

IT MUST BE A TROLL.

 

[ray]

This is a valid high-level argument: success is much more than
avoiding failure. Even glaring failures can be immaterial.

However, I am reading in ALA where details are very relevant so
I feel compelled to offer some of the many:


1) A Linux distro is _not_ the kernel. distros come and go,
the kernel is eternal

2) Much greater code review has been done for OpenBSD. It has
not lead to run-away success outside of its' domain.

3) Using a desktop/user distro for a "critical support system"
is unlikely to be successful except for "non-traditional"
definitions of "critical"

Would a real time monitoring system in a major DOD test and evaluation
environment qualify? I think so. And the ones I was familiar with before I
retired three years ago relied on Unix and Linux.

 

[Robert Redelmeier]

Would a real time monitoring system in a major DOD test and evaluation
environment qualify? I think so. And the ones I was familiar with
before I retired three years ago relied on Unix and Linux.

Certainly it would qualify as "critical". And I don't doubt
Unix and other Linux-like systems could pass.

All I'm trying to say is that I doubt an out-of-the-box,
install everything _user_ distro like Ubuntu would pass. Just
think of all the kernel modules. Redhat, Debian, Slackware or
even a correctly stripped Ubuntu would be necessary, preferably
with a kernel customized for the hardware. No X-server, XDM or
other eye candy will necessarily be reliable on all hardware.

-- Robert

 

[Evenbit]

Ah... theory. Leaves a nice warm feeling, doesn't it?

.... and leads to my trade-mark trait of "going off half-cocked!"

Three potential solutions to fix your unsound comments.

1) Re-read those books.
2) Get more modern/informative books.
3) Try a little practical implementation so you can see why it is so
foolish to back such inconsistent theories or potential
misunderstandings.

Luckily, just re-reading them was enough to convince me of my error.
My confusion was due to putting too much emphasis on the fact that
blocks always contain pages that are assigned to contiguous regions of
a process' address space. It is true that it is possible to fragment
(make a mess of) the process' address space, but this should only
happen due to extremely bad code or via intentional effort on the part
of the programmer. Even then, I suspect that any "fragmentation wall"
is purely theoretical because your process would be stopped long
before it could be hit.

Also, if you write an application that is extremely memory-hungry, you
need to scrap everything and go back to the flow-charts for an
entirely different design.

Nathan.

 

[Malcolm McLean]

Unless of course some brain-damaged library designer has arranged that
malloc never, ever returns NULL, so that even properly-written
applications can't tell whether a request succeeded or not. If ever there
was a decision that favoured the mediocre over the competent, that's the
one.

That's me.
Check out xmalloc(), a malloc that never returnsNULL, on my website.

 

[$)CHarald van D)&k]

That's me.
Check out xmalloc(), a malloc that never returnsNULL, on my website.

Your xmalloc aborts (by default) when no memory is available. I believe
Mr. Heathfield was referring to a particular malloc implementation that
returns a non-null pointer even when no memory is available, causing
either that program, or some semi-random other program to crash at some
later time when the memory gets used, without any way for the program to
handle the error.

 

[CBFalconer]

Your xmalloc aborts (by default) when no memory is available. I believe
Mr. Heathfield was referring to a particular malloc implementation that
returns a non-null pointer even when no memory is available, causing
either that program, or some semi-random other program to crash at some
later time when the memory gets used, without any way for the program to
handle the error.

I doubt it. He is probably referring to just what Mr. McLean has
apparently done, i.e. automatic program abort. Now the program has
no way of detecting that memory is no longer available, and cannot
take remedial steps.

 

[Richard Heathfield]

??Harald van D??k said:

I believe
Mr. Heathfield was referring to a particular malloc implementation that
returns a non-null pointer even when no memory is available [...]

Mr van Dijk believes correctly.

 

[Richard Heathfield]

CBFalconer said:

I doubt it.

You doubt wrong.

He is probably referring to just what Mr. McLean has
apparently done, i.e. automatic program abort. Now the program has
no way of detecting that memory is no longer available, and cannot
take remedial steps.

No, it is easy to solve the problems caused by Mr McLean's function, by the
simple expedient of not using it.

It's a lot harder to avoid malloc.

 

[Keith Kanios]

... and leads to my trade-mark trait of "going off half-cocked!"

There's nothing to prove around here, only knowledge to gain and
eventually give back.

Luckily, just re-reading them was enough to convince me of my error.
My confusion was due to putting too much emphasis on the fact that
blocks always contain pages that are assigned to contiguous regions of
a process' address space. It is true that it is possible to fragment
(make a mess of) the process' address space, but this should only
happen due to extremely bad code or via intentional effort on the part
of the programmer. Even then, I suspect that any "fragmentation wall"
is purely theoretical because your process would be stopped long
before it could be hit.

Also, if you write an application that is extremely memory-hungry, you
need to scrap everything and go back to the flow-charts for an
entirely different design.

Nathan.

There you go

 

[Richard Bos]

That's me.

To a tee.

Check out xmalloc(), a malloc that never returnsNULL, on my website.

A malloc() replacement that never returns a null pointer must eventually
resort to doing something much worse.

Richard

 

[Malcolm McLean]

A malloc() replacement that never returns a null pointer must eventually
resort to doing something much worse.

It's good for medium-critical applications.
If you request a small amount of memory then it is unlikely that the user
cannot close down some other application to release it. Alternatively the
callback could release an emergency store. However then it would have to
post a quit message, because of the danger that user might simply continue.

If you request a massive amount of memory that's probably because of a bug
or malicious input, so it is unclear what to do in that situation anyway.

In very critical applications, you will want to recover gracefully from a
memory allocation failure. This adds very substantially to the costs of
writing the code, and is also difficult to test. However if the program must
under no circumstances terminate, this is what you must do. xmalloc()
supposes that termination can be used as a last / first resort, depending on
the program.

 

[Richard Bos]

It's good for medium-critical applications.
If you request a small amount of memory then it is unlikely that the user
cannot close down some other application to release it. Alternatively the
callback could release an emergency store. However then it would have to
post a quit message, because of the danger that user might simply continue.

If you request a massive amount of memory that's probably because of a bug
or malicious input, so it is unclear what to do in that situation anyway.

In very critical applications, you will want to recover gracefully from a
memory allocation failure. This adds very substantially to the costs of
writing the code, and is also difficult to test. However if the program must
under no circumstances terminate, this is what you must do. xmalloc()
supposes that termination can be used as a last / first resort, depending on
the program.

Tell me, have you ever pasted a large graphic into a report?

Richard

 

[Malcolm McLean]

Tell me, have you ever pasted a large graphic into a report?

You shouldn't use the xmalloc() handler as part of the "normal" flow control
of the program.
Let's say we are allocating an image of 1 million bytes. At the same time we
allocate a hundred byte string to hold the image name. Let's also say that
the user experiences one allocation failure per day of using the program, on
average.
A simple calculation will show that, assuming 50 working weeks of 5 days
each, he can expect the failure to be in the hundred byte allocation about
once every 40 years. For the vast majority of "lose a day's work" type
critical applications, that sort of failure rate is acceptable. it is much
lower than the chance of a hardware failure in the same time. On the other
hand one failure a day isn't acceptable, so we've got to handle it rather
better. But the failure-handling routines tend to add considerable
complexity to code, and it's hard to test.
The point of xmalloc() is that even if the allocation does fail, we can
choose to give the user a chance to get some more memory. However we don't
have special error-handling recovery code. It's simply not worth coding and
testing - and they tend to be hard to test - custom error handlers for such
rare problems, unless the future of the business is at stake if we have to
crash. It's a misallocation of scare programming resources.

Note that even if the once in forty years problem does come up, we can still
give the user the chance to close down some applications. If he can't get a
hundred bytes from the system, anywhere, it is likely that he'll have to
reboot anyway.