In my FreeBSD source tree this part of calloc() is *before* malloc() has
been called at all.
Indeed.
Obviously, the purpose of the particular if check is to avoid attempting
to malloc() a huge amount of space, that cannot be satisfied anyway.
Actually, it is more significant than that.
The two arguments to calloc() both have type "size_t", which is an
unsigned type. Suppose (for argument's sake, or because it also
happens to be true for the Intel-based FreeBSD anyway) that size_t
is an alias for a 32-bit unsigned integer type.
Suppose further that the parameters to calloc() are 10000000 (10
million) and 65536 -- i.e., 0x00989680 and 0x00010000. (Since
multiplication is commutative, it does not really matter which of
these is "num" and which is "size".) What is the product, num*size?
Clearly it *should* be 655360000000, or 0x9896800000, but this
exceeds 32 bits. Due to the nature of unsigned arithmetic, the
result will be reduced modulo $2^{32}$ (TeX notation), giving
0x96800000 or 2524971008.
Chances are this particular malloc() will then fail, but if it
succeeds -- which is more likely if I were to fiddle the numbers
above to come up with smaller final result -- the calloc()
routine will then bzero() the same amount of memory, and finally
return that pointer to its caller -- which will expect to have
much more memory than it actually does.
This could open a security hole, depending on the application in
question and what it does with the memory it believes it has.
The OP (identified only as "brian" above) did ask why this is done
even "if no more than one object is being allocated", i.e., if num
< 2. Since num is an unsigned type (size_t), this means either
num==0 or num==1. If num==0, the first part of the test -- "num
&&" -- fails and the "if" does nothing. So the work is only done
"questionably" if num==1 (and then if size > 0). In this case,
SIZE_T_MAX / num is SIZE_T_MAX / 1 which is clearly still SIZE_T_MAX,
and since "size" also has type size_t, the test SIZE_T_MAX < size
will fail. (The varable "size" is always less than or equal to
SIZE_T_MAX.) So in this case, the test is redundant -- but so
what?
One could argue that the code might be a hair faster for the num==1
cases if the test read:
if (num > 1 && size && SIZE_T_MAX / num < size)
which is probably true, but probably also not really very interesting
-- not unless profiling shows calls to calloc() with num==1 being
high on the time-consumption list.