Experiment: functional concepts in C

[Stefan Ram]

int withFile(const char *fileName,
const char *mode,
int (*user)(FILE *))
{
FILE *fh;
int err;

fh = fopen(fileName, mode);
if (fh == NULL) return 1;

err = user(fh);
fclose(fh);
return err;
}

Sometimes, an additional fourth parameter for a client data
object (as in »err = user( fh, data )«) might come in handy.

The function »user« should not give a meaning to the result
value »1« as this could be confused with the indication for
»fh == NULL«. When such functions are nested, this can become
more complicated, so either a globel return value meaning table
is needed or each function needs to convert error results of
called functions (deleting some information).

 

[Stefan Ram]

Of course you can always write an interpreter for a richer language.
That's not really 'doing it in C'.

Yes, that was also my point.

The assignment to »teach C« usually is /not/ fulfilled by
teaching to create functions as

gmap(gmap(gmap(fun3(aLzy2(__1), aLzy(__1), __1), zs), ys), xs)

although this is a part of a C program.

But to teach C, one usually writes programs using only those
concepts of the language that come »out of the box«, i.e.,
are available from an (from every) implementation of C
according to ISO/IEC 9899:1999 (E).

For example, I would teach Linux system calls in C only iff
the assignment is not »teach C«, but »teach Linux system
programming in C«. However, I /do/ include some examples
showing how third party libraries can be employed in own
projects, but otherwise, I do not make the source code
depending on them.

 

[Ersek, Laszlo]

if fclose() doesn't work why should fflush()?

It's not that fflush() should work. My point is that you're not done and
not ready to relinquish control to the RTS until you flush your buffered
writes, because flushing is when writing actually happens. Until that
point, you just declare your intent to write. If the RTS notices that
you have left some data buffered and it's unable to flush them, it has
no way to tell you.

Cheers,
lacos

 

[Michael Foukarakis]

It behaves "as if" it were created as an "auto" and copied on
assignment.


Whoever takes a copy cleans up their own copy.

Why? Wasn't it created "as if" it were an automatic variable? Why the
need to clean up after automatic variables? </
angstyfaceofnewbieterror>

RAII.  They're automatically destroyed once they're used.

Let's take a moment to think about that. I allocated a Matrix with
new. You said it behaves "as if" it were an automatic variable - why
would that be? Let's skip that question for a moment and let's think
about who destroys the Matrix; it must be someone who allocated it
with new, or someone that received a reference to it (explicitly
stated conventions notwithstanding etc.), because, as far as I can
tell, there is no automatic memory management in C++. Perhaps I am
missing something in your train of thought, though.

It is "as if" it is, yes.


Only if the compiler cannot optimise the assignment away, while still
behaving "as if" there were lots of copies.  Most compilers (even C
compilers) don't return large structures on the stack or in registers,
but use a hidden reference argument, so most compilers elide the
copies.   See RVO and NRVO.
Of course, if you implement operator+=, you can elide those copies
yourself.


Non-sequitor.

It'd be one if I were talking about crab ambulances. But, the pigeons
are still sleeping.

Whoop-de-do.  I've seen literally thousands of complaints about almost
everything.  People like complaining.  Can you name me a language
about which vociferous angry people do not frequently complain?

Well, complaints usually arise when people with a certain mindset (for
instance, systems people) try to bring that mindset into other
contexts (for instance, web development). You know, the same people
that write C-like Perl. That aside, I've never heard a lot of
complaining about Python. Python enthusiasts seem to be just that -
enthusiastic.

Not massively.  You're not obliged to use every feature of the
language, and if you try to in C++, it will drive you insane.  If
you're talking about doing heavily-templated-metaprogramming without
understanding RVO, you might be trying to run before you can walk. But
good design, a bit of care and a lot of restraint will see you through.

You are right in a certain degree. My pov is that you cannot do
heavily templated metaprogramming without running into severe language
restrictions (Understanding RVO? Really? Do I have to be familiar with
compiler optimizations to write correct code? Wow..), and in the end,
people use a small subset of C++ (which is a pity) and sleep well at
night. Those that will try to use the full power of the language (and
in theory it SHOULD be very powerful) will waste their time chasing
overload resolution ambiguities. I don't even care.

 

[ld]

  Yes, that was also my point.

There is a strong difference between an interpreter, a preprocessor
and an embedded DSL.

  The assignment to »teach C« usually is /not/ fulfilled by
  teaching to create functions as

gmap(gmap(gmap(fun3(aLzy2(__1), aLzy(__1), __1), zs), ys), xs)

  although this is a part of a C program.

The statement was that closures cannot be built/used in C. I agree
that explaining the code above is beyond an advanced course on C, but
nevertheless it is possible in C.

 

[Richard Bos]

Operator overloading and constructors/destructors, which are additions
that Jacob has vehemently opposed in the past.

No, we have always been at war with Eastasia.

Richard

 

[Stefan Ram]

The statement was that closures cannot be built/used in C. I agree
that explaining the code above is beyond an advanced course on C, but
nevertheless it is possible in C.

When given /only/ a C implementation, one needs a lot of
effort to build such a library and then use it. One does not
need this effort when using a language where these features
are already part of the language. So, yes, it is possible in
C, but might be more effort.

You might say that one does not need to implement those
features oneself, but can use a third-party library »L« for it.
But then, the basis for development would not be »C«, but »C+L«.
Nothing is wrong with using »C+L« as a basis, but it should
just not be called »C«, but »C+L«.

 

[Ertugrul Söylemez]

Commenting there is so painful (the comment editor embedded in the
page doesn't recognise arrow keys!) I'll reply here instead!

As soon as my own blog software is up and running, I'll go away from
Blogger anyway, so I really prefer comments here.

It's an interesting approach, and I can certainly see its value in a
structure that requires a lot of specialised initialisation and
destruction. You could replace:

struct foo *foop = Create_Foo_Structure();
lots-and-lots-of-code-using foop
Release_Foo_Structure(foop)

with

withFoo(dofoopstuff);

but, of course, you have to create dofoopstuff(struct foo *foop)
before you can do that, and in that case writing:

struct foo *foop = Create_Foo_Structure();
dofoopstuff(foop);
Release_Foo_Structure(foop)

does the same job, and you aren't likely to forget to release it when
it's only two lines from creation.

That's a general problem of the way C programmers think. Yes, if you
need it only once, you are faster just writing it out. But the point
here is not efficiency, it's safety! It's specifically that "you aren't
likely to forget" part of your reasoning, which is the big problem.
With that support function at hand, you _will not_ forget, and that's a
great advantage.

And that's where I think this fails. If this is the only benefit you
get (in C obviously - I know some of the stuff you can do in a better
framework) then it doesn't seem to give anything more than a
programming discipline that says:
- remove anything that uses an allocated structure (including a file
pointer) into a separate function.
- call the function immediately between allocation and release of the
structures, and don't place /any/ other code in there.

Still a neat idea, but I'm not convinced it normally has any practical
value.

This is the only benefit, yes. But it's not a small benefit. It will
help make C code safer. It's just that I'm afraid someone needs to drop
this into a somewhat large library. Otherwise nobody will use it.


Greets
Ertugrul

 

[Nick]

As soon as my own blog software is up and running, I'll go away from
Blogger anyway, so I really prefer comments here.



That's a general problem of the way C programmers think.

Rather, I think, with the way people think when they are programming in
C. Because of the way C works we are limited to just doing this.

Yes, if you
need it only once, you are faster just writing it out. But the point
here is not efficiency, it's safety! It's specifically that "you aren't
likely to forget" part of your reasoning, which is the big problem.
With that support function at hand, you _will not_ forget, and that's a
great advantage.

In this discussion I'm not actually that bothered about efficiency -
either in terms of programmer effort or execution. I'm more bothered
about benefits traded off against drawbacks.

To me, there's a big drawback here in that you end up splitting what may
well conceptually be a single piece of code into multiple functions. If
you could stick literal functions in the call, for example, you could
put three line sequences that need a file pointer inside the "with
pointer" call. But in C you can't.

So you are going to end up with your logical block of code scattered
into lots of little bits.

That to me seems a big drawback. You say - correctly - that you are
less likely to forget to free the resource. I sort of agree, but - in
my C programming experience - you tend to forget to release the resource
(or, far worse, release it twice) when you have the code all in one
block.

This is the only benefit, yes. But it's not a small benefit. It will
help make C code safer. It's just that I'm afraid someone needs to drop
this into a somewhat large library. Otherwise nobody will use it.

And this is where I disagree. Once you've done the breaking the code
into little nuggets then wrapping each with "acquire" and "release" is
going to be simple. You really aren't going to forget to do that
sequence, or put two releases in there. So the rather clumsy syntax
gains you nothing.

I summary, I think that the limitations of C mean that this doesn't give
enough benefits. But you've got me thinking about a hideous generic
"WITH_RESOURCE" macro that takes an acquire function, a function to
call, a release function, and a pile of arguments!

 

[Ertugrul Söylemez]

In this discussion I'm not actually that bothered about efficiency -
either in terms of programmer effort or execution. I'm more bothered
about benefits traded off against drawbacks.

To me, there's a big drawback here in that you end up splitting what
may well conceptually be a single piece of code into multiple
functions. If you could stick literal functions in the call, for
example, you could put three line sequences that need a file pointer
inside the "with pointer" call. But in C you can't.

So you are going to end up with your logical block of code scattered
into lots of little bits.

That to me seems a big drawback. You say - correctly - that you are
less likely to forget to free the resource. I sort of agree, but - in
my C programming experience - you tend to forget to release the
resource (or, far worse, release it twice) when you have the code all
in one block.

Indeed, there is little syntactic support, but ...

And this is where I disagree. Once you've done the breaking the code
into little nuggets then wrapping each with "acquire" and "release" is
going to be simple. You really aren't going to forget to do that
sequence, or put two releases in there. So the rather clumsy syntax
gains you nothing.

I summary, I think that the limitations of C mean that this doesn't
give enough benefits. But you've got me thinking about a hideous
generic "WITH_RESOURCE" macro that takes an acquire function, a
function to call, a release function, and a pile of arguments!

.... that idea is actually great. Yes, a macro should work very well
here. I haven't thought of that.


Greets
Ertugrul

 

[Pascal J. Bourguignon]

Yes, the whole point of my post is that C usually does
/not have/ a GC. So I infer that functional programming,
therefore, is difficult or nearly impossible (beyond a
certain level of complexity) in C.

When the Lisp Machines had only a few megaconses of memory, if not half
a megacons, I've been told they could run for a week without needing a
garbage collection, and people just rebooted them once a week to avoid
it.

Well, next month I'll buy a PC, a PERSONAL computer with 24 GB ram, or
3 Gconses. I guess it could run without a garbage collection for a
month!

Perhaps today, for short lived C processes, you just don't have to care
about free anymore. Just malloc, and exit.

 

[Ertugrul Söylemez]

When the Lisp Machines had only a few megaconses of memory, if not
half a megacons, I've been told they could run for a week without
needing a garbage collection, and people just rebooted them once a
week to avoid it.

Well, next month I'll buy a PC, a PERSONAL computer with 24 GB ram, or
3 Gconses. I guess it could run without a garbage collection for a
month!

Perhaps today, for short lived C processes, you just don't have to
care about free anymore. Just malloc, and exit.

This reasoning is used by many PHP programmers today. It's one reason
why PHP programs are so error-prone and why PHP errors are often hard to
find. There is no excuse for unfreed resources, particularly in
languages, which are unsafe by nature, like C and PHP. It _will_ get
you into trouble sooner or later.

Note that in both C and PHP most results are computed solely through
side-effects and are never obtained in a direct fashion. It is very
important that you get these side effects under control, even if your
process is short-lived. My proposal is one way of doing this.

The better way is to avoid using side effects altogether, but that's
close to impossible in C (and PHP).


Greets
Ertugrul

 

[santosh]

Pascal J. Bourguignon wrote:



Perhaps that's the reasoning used by the Firefox guys. When running
Win32, I choose Firefox rather than IE, but that's becoming a harder
decision than it should be. The other day, I peeked at Task Manager and
found that Firefox (which I rarely leave running for long because I know
it's a hog) had grabbed well over half a gigabyte of memory. I rarely
have more than one window open, and rarely more than one (perhaps two or
three, and very occasionally up to a dozen) tabs in that window. Half a
gigabyte. It's just ludicrous.

Are you sure that's physical memory? During normal use (similar to
what you describe), Firefox does grab over 500 megabytes of virtual
memory, but usually only about a couple of hundred Mb of physical
memory. Atleast, that's the usage pattern for me here. And I'm running
Linux, but I suspect memory usage is quite similar in Windows too.

And it slows everything else down,
because less available RAM means more swapping, and more swapping means
more disk access.

I notice though that all popular graphical browsers have roughly
comparable memory consumption. Certainly, in my experience, Firefox,
SeaMonkey, Galeon, Chrome. Opera tends to consume about 50 Mb less
than these.

Stupid stupid stupid. Don't follow their bad example. Write code
properly. *Please*.

I wouldn't single out Firefox though. The fact that all major
graphical browsers consume roughly similar amounts of memory for
similar use seems to show that either there's a good reason or that
fine grained resource management gets unmanageably complicated in very
large software projects.

It'd be interesting to see what difference a garbage collector does to
a program like Firefox (assuming it doesn't already have one inside.)
It would give us a clue as to whether the memory hogging is due to
tangible requirements or just accumulation of expired objects.

 

[Ersek, Laszlo]

Perhaps today, for short lived C processes, you just don't have to care
about free anymore. Just malloc, and exit.

That seems to be a bit negligent even for programs that allocate memory
frugally; for me it encourages a style where you don't know something
about your program's state that you should know in general.

And of course there are short lived C programs that are a bit hungry.
I've just repeated a 64-bit decompression test on the NIIFI's computer,
described under [0], with 55 worker threads. (The input file is
8,457,341,456 bytes big.) lbzip2 ran for 1 minute and 34.81 seconds, so
it could reasonably be called a short lived process. Its peak allocation
was 1,349,156,524 bytes. Its *cumulative* allocation was 178,286,146,620
bytes.

Cheers,
lacos

[0] http://lacos.hu/lbzip2-scaling/scaling.html

 

[Ersek, Laszlo]

Are you sure that's physical memory?

It better be. I configure my system that way.

During normal use (similar to
what you describe), Firefox does grab over 500 megabytes of virtual
memory, but usually only about a couple of hundred Mb of physical
memory.

And that's an invitation for a stab in the back by the OOM killer.
Rather waste sincerely and honestly than speculate.


(Yes, I know this was written by Richard.) Semi-modern workstations
should run without swap, in my opinion; "a few G or RAM should be enough
for everyone" Really, suppose you have 2G RAM, and some processes
spill over into swap. The random-like memory access patterns of several
processes are now reflected by the head movements of your consumer-grade
hard disk. (The I/O scheduler, if any, can help only so much.) You won't
be able to bring up a terminal or task manager to find / suspend / kill
the offender(s) in reasonable time.

In a desktop situation where swap would help, swap is no help. Most of
the time I like a clean and quick "out of memory, exiting" message more
than waiting for several minutes on the disk (and probably getting the
same message in the end).

I have no experience with solid-state drives, but consumer-grade thumb
drives etc have *catastrophic* random-write performance, especially the
bigger ones -- they are worse for swapping than traditional hard disks.

I notice though that all popular graphical browsers have roughly
comparable memory consumption. Certainly, in my experience, Firefox,
SeaMonkey, Galeon, Chrome. Opera tends to consume about 50 Mb less
than these.

I suppose browsers have to handle lots and lots of small objects, ie.
objects that don't deserve their own mmap()-ed pages in the default
malloc configuration. This likely leads to hellish heap fragmentation
and a huge heap VMA. It would be interesting to see how memory usage is
affected after a long browsing session, when one opens a pristine
window, closes everything else, stops all downloads, and clears all
history and caches. In that case, even the heap VMA should be trimmed.

Cheers,
lacos

 

[Nobody]

(Yes, I know this was written by Richard.) Semi-modern workstations should
run without swap, in my opinion;

You should have enough swap for things which should get swapped out and
(normally) stay swapped out.

Many programs initialise a lot of data at startup then never touch it
again unless you use that particular feature. There's no point having such
data locked into RAM when there are better ways to use it.

Similarly, modern desktop systems tend to run a lot of programs which are
"running" for as long as you're logged in but seldom see actual use.

 

[Willem]

Richard Heathfield wrote:
) santosh wrote:
)> Are you sure that's physical memory?
)
) Well, of course your asking the question makes me doubt the answer, and
) I lack the time to check in more detail right now, BUT...
)
) > During normal use (similar to
)> what you describe), Firefox does grab over 500 megabytes of virtual
)> memory, but usually only about a couple of hundred Mb of physical
)> memory.
)
) *ONLY* a couple of hundred megabytes? Can you hear yourself?

<snip>

Isn't that just the in-memory cache though ?
Could very well even be a config option.


SaSW, Willem
--
Disclaimer: I am in no way responsible for any of the statements
made in the above text. For all I know I might be
drugged or something..
No I'm not paranoid. You all think I'm paranoid, don't you !
#EOT

 

[Andrew Poelstra]

Richard Heathfield wrote:
) santosh wrote:
)> Are you sure that's physical memory?
)
) Well, of course your asking the question makes me doubt the answer, and
) I lack the time to check in more detail right now, BUT...
)
) > During normal use (similar to
)> what you describe), Firefox does grab over 500 megabytes of virtual
)> memory, but usually only about a couple of hundred Mb of physical
)> memory.
)
) *ONLY* a couple of hundred megabytes? Can you hear yourself?

<snip>

Isn't that just the in-memory cache though ?
Could very well even be a config option.

That might explain why it also takes ten seconds to load, while
Opera takes four and Chrome just over one.

IME since 2.1 (IIRC) Firefox has been progressively slower and
bloatier. The latest version was a slight uptick, but not much.

 

[Phil Carmody]

Many programs initialise a lot of data at startup then never touch it
again unless you use that particular feature. There's no point having such
data locked into RAM when there are better ways to use it.

Similarly, modern desktop systems tend to run a lot of programs which are
"running" for as long as you're logged in but seldom see actual use.

Hey, don't forget about mobile phones - they do that too!

And your two paragraphs are linked. Because there's so much
unnecessary crap happening at program startup, one can no
longer afford to start them on demand - they need to be pre-
started. Which means that they have to hang around while you're
not using them.

Phil

 

[Andrew Poelstra]

There is generally no need to free resources just before
program exit, because the operating system will do that

What basis do you have for this absurd claim...

for you - a program which spends a bunch of time freeing
every allocated object and then exits has just wasted all

....or this one? The OS, if it frees your memory, needs just
as much CPU time as your code would, had you chosen to make
it portable to machines who are not your mother.