Memory footprint of a structure of structures

[Jean-Michel Hautbois]

Hi,

I have a (big) structure, which contains other structures with
(sometimes) big buffers.

Something like :

#define BUF_SIZE 65536
#define BUF_SIZE2 32768

typedef struct {
unsigned char buffer1[BUF_SIZE];
unsigned char buffer2[BUF_SIZE];
} t_buffer_head;

typedef struct {
unsigned char buffer1[BUF_SIZE2];
unsigned char buffer2[BUF_SIZE2];
} t_buffer_subhead;

typedef struct {
t_buffer_head header;
t_buffer_subhead subheader;
} t_buffer_s;

In my use case, I have >100 structures of t_buffer_something and one
big structure of these structures.
I would like, at compile time ideally, or by a static analysis script,
to know the footprint in memory of the big structure (here,
t_buffer_s).

Something like :

t_buffer_s (196608 bytes) :
|__t_buffer_head (131072)
| |__buffer1 (65536)
| |__buffer2 (65536)
|__t_buffer_subhead (65536)
|__buffer1 (32768)
|__buffer2 (32768)

Of course, in case the structure contains pointers, it would likely
indicate it whith a (???) because not knowing the malloc size
associated.

This would be the ideal format, but any approaching solution is
interesting.

Thanks in advance for your advices !
Regards,
JM

 

[James Kuyper]

Hi,

I have a (big) structure, which contains other structures with
(sometimes) big buffers.

Something like :

#define BUF_SIZE 65536
#define BUF_SIZE2 32768

typedef struct {
unsigned char buffer1[BUF_SIZE];
unsigned char buffer2[BUF_SIZE];
} t_buffer_head;

typedef struct {
unsigned char buffer1[BUF_SIZE2];
unsigned char buffer2[BUF_SIZE2];
} t_buffer_subhead;

typedef struct {
t_buffer_head header;
t_buffer_subhead subheader;
} t_buffer_s;

In my use case, I have >100 structures of t_buffer_something and one
big structure of these structures.
I would like, at compile time ideally, or by a static analysis script,
to know the footprint in memory of the big structure (here,
t_buffer_s).

Any capability to calculate such things at compile time would have to be
a feature of the particular compiler you're using; the C standard
mandates no such feature. Offhand, I don't know of any such feature
provided by any compiler I use, but then I don't use a lot of different
compilers. Without specifying which compiler you want to use, there's no
way anyone can answer that question for you.

buffer1 and buffer2 have sizes that can be determined just by static
analysis of the code, However, the size of anything other than character
variables and character arrays can vary from one implementation of C to
another. Therefore, any such static analysis tool will have to be
associated with a particular compiler. Again, the question cannot be
answered without specifying the compiler you want to use.

If you do have a particular compiler in mind, for best results you
should post your question in a forum devoted to that particular compiler.

Something like :

t_buffer_s (196608 bytes) :
|__t_buffer_head (131072)
| |__buffer1 (65536)
| |__buffer2 (65536)
|__t_buffer_subhead (65536)
|__buffer1 (32768)
|__buffer2 (32768)

Something that would work perfectly fine, and on all compilers, would be
to write a program that contains a declaration for your struct types,
and prints out the sizes using sizeof(), in whatever format you want.
That seems like such a simple solution that I presume you've already
considered it and have decided it's not convenient? You'll need to write
new code to perform the printout each time you create a new struct or
change the members of a struct, and you'll need to recompile it each
time the type or dimensions of a member changes; I suppose that could be
inconvenient.

 

[Jorgen Grahn]

Hi,

I have a (big) structure, which contains other structures with
(sometimes) big buffers.

Something like : ....
typedef struct {
t_buffer_head header;
t_buffer_subhead subheader;
} t_buffer_s;

In my use case, I have >100 structures of t_buffer_something and one
big structure of these structures.
I would like, at compile time ideally, or by a static analysis script,
to know the footprint in memory of the big structure (here,
t_buffer_s).

Something like :

t_buffer_s (196608 bytes) :
|__t_buffer_head (131072)
| |__buffer1 (65536)
| |__buffer2 (65536)
|__t_buffer_subhead (65536)
|__buffer1 (32768)
|__buffer2 (32768)

Of course, in case the structure contains pointers, it would likely
indicate it whith a (???) because not knowing the malloc size
associated.

Or if it's malloced at all, or if a million t_buffer_s structs point
to the same tiny malloced thing ...

This would be the ideal format, but any approaching solution is
interesting.

In the past, I have used gcc and (I think) its option to generate
STABS debug information. This format is quite readable, although it
gives you everything in bits, not bytes.

There used to be a utility which came with Perl and wrapped that quite
nicely. I can't remember its name though, but I think it was written by
Tom Christiansen. *googles* Ok, it's called "pstruct" or "c2ph".

/Jorgen

 

[Philipp Klaus Krause]

Am 23.11.2011 12:33, schrieb James Kuyper:

Hi,

I have a (big) structure, which contains other structures with
(sometimes) big buffers.

Something like :

#define BUF_SIZE 65536
#define BUF_SIZE2 32768

typedef struct {
unsigned char buffer1[BUF_SIZE];
unsigned char buffer2[BUF_SIZE];
} t_buffer_head;

typedef struct {
unsigned char buffer1[BUF_SIZE2];
unsigned char buffer2[BUF_SIZE2];
} t_buffer_subhead;

typedef struct {
t_buffer_head header;
t_buffer_subhead subheader;
} t_buffer_s;

In my use case, I have >100 structures of t_buffer_something and one
big structure of these structures.
I would like, at compile time ideally, or by a static analysis script,
to know the footprint in memory of the big structure (here,
t_buffer_s).

Any capability to calculate such things at compile time would have to be
a feature of the particular compiler you're using; the C standard
mandates no such feature.

How about sizeof?

Philipp

 

[James Kuyper]

Am 23.11.2011 12:33, schrieb James Kuyper:

How about sizeof?

The C standard does mandate the sizeof operator, and it can be used to
create a program which, at run-time, will generate output like that
which he's asking for; I said as much in a later part of my message
which you snipped.

The C standard does not mandate the existence of any feature that would
produce such output at compile time. The only output that normally
appears at compile time and is mandated by the C standard is diagnostic
messages, the text of which is not specified by the standard, nor under
the control of the developer - with the exception of the #error
directive. However, the #error directive is executed during phase 4;
sizeof expressions cannot be evaluated until phase 7, so the value of
such expressions cannot appear in #error output.

 

[Ark]

The C standard does mandate the sizeof operator, and it can be used to
create a program which, at run-time, will generate output like that
which he's asking for; I said as much in a later part of my message
which you snipped.

The C standard does not mandate the existence of any feature that would
produce such output at compile time. The only output that normally
appears at compile time and is mandated by the C standard is diagnostic
messages, the text of which is not specified by the standard, nor under
the control of the developer - with the exception of the #error
directive. However, the #error directive is executed during phase 4;
sizeof expressions cannot be evaluated until phase 7, so the value of
such expressions cannot appear in #error output.

Are you sure you are not mixing up sizeof type and sizeof object?

 

[Ark]

Vincenzo Mercuri ha scritto:
[...]

I wonder how this could be useful if not in some "compile-time assert-
like macros". There is some feature like this in the "gnulib" library,
something like the "_Static_assert" specified by the C1X draft.
Take a look: http://goo.gl/4lMfK

Oh, it looks like static assertions are supported in gcc 4.6 and later
http://gcc.gnu.org/gcc-4.6/changes.html#c

It's been elaborated in this NG some year ago:
#define static_assert(expr) extern char dummy_array[(expr)?:1:-1]

 

[Philip Lantz]

Are you sure you are not mixing up sizeof type and sizeof object?

In what way do you think he might be mixing them up? Do you think that
sizeof type is evaluated in a different phase from sizeof object?

 

[Jean-Michel Hautbois]

Jean-Michel Hautbois ha scritto:
[...]> In my use case, I have>100 structures of t_buffer_something and one

big structure of these structures.
I would like, at compile time ideally, or by a static analysis script,
to know the footprint in memory of the big structure (here,
t_buffer_s).

[...]

I was not sure of what you meant by "footprint in memory". Then I
searched on wikipedia and found: "Memory footprint refers to the amount
of main memory that a program uses or references while running".

How can you have at compile time the amount of memory that a program
uses or references while running?

I'm confused, but the fact that you are not interested in memory
dynamically allocated (i.e. "malloced") makes me think that you only
need to know the total size of the structures.

Yes, I didn't know how to call it . It is the static size of the
structure that interests me.

I wonder how this could be useful if not in some "compile-time assert-
like macros". There is some feature like this in the "gnulib" library,
something like the "_Static_assert" specified by the C1X draft.
Take a look:http://goo.gl/4lMfK

Thanks, I didn't thought about that stuff !
I will think about a way to use it...
JM

 

[Philipp Klaus Krause]

Am 23.11.2011 21:41, schrieb James Kuyper:

The C standard does mandate the sizeof operator, and it can be used to
create a program which, at run-time, will generate output like that
which he's asking for; I said as much in a later part of my message
which you snipped.

The C standard does not mandate the existence of any feature that would
produce such output at compile time. The only output that normally
appears at compile time and is mandated by the C standard is diagnostic
messages, the text of which is not specified by the standard, nor under
the control of the developer - with the exception of the #error
directive. However, the #error directive is executed during phase 4;
sizeof expressions cannot be evaluated until phase 7, so the value of
such expressions cannot appear in #error output.

Well, sizeof, unless applied to a variable-length array, yields an
integer constant. Which AFAIK, can be used e.g. as array bound. That's
how I understood the OP's "would like, at compile time ideally, […], to
know the footprint in memory" requirement.

Philipp

If, on the other hand, the OP just wants to know personally about the
size of objects in memory (as opposed to using the result in the program
as above): Many linkers have a command-line option to give the desired
information.

 

[Jorgen Grahn]

Jean-Michel Hautbois ha scritto:
[...]

In my use case, I have>100 structures of t_buffer_something and one
big structure of these structures.
I would like, at compile time ideally, or by a static analysis script,
to know the footprint in memory of the big structure (here,
t_buffer_s).

[...]

I was not sure of what you meant by "footprint in memory". Then I
searched on wikipedia and found: "Memory footprint refers to the amount
of main memory that a program uses or references while running".

How can you have at compile time the amount of memory that a program
uses or references while running?

I'm confused, but the fact that you are not interested in memory
dynamically allocated (i.e. "malloced") makes me think that you only
need to know the total size of the structures.

It's quite common IME to want to get a feel for the size of structs.
If the nesting level is deep and you didn't write the code yourself,
it can be time-consuming and error-prone to figure out manually if
something takes 1000 bytes or 50,000 bytes ...

So it's not really an odd question.

/Jorgen

 

[BartC]

It's quite common IME to want to get a feel for the size of structs.
If the nesting level is deep and you didn't write the code yourself,
it can be time-consuming and error-prone to figure out manually if
something takes 1000 bytes or 50,000 bytes ...

You write a one-line program, or one-line extension, to simply print the
static size, of the top-most struct.

But OP wants the sizes to be displayed in a tree-format, without the effort
of writing, and having to maintain in parallel, a long sequence of
printf/sizeof lines.

 

[Jean-Michel Hautbois]

You write a one-line program, or one-line extension, to simply print the
static size, of the top-most struct.

But OP wants the sizes to be displayed in a tree-format, without the effort
of writing, and having to maintain in parallel, a long sequence of
printf/sizeof lines.

Exactly And AFAIK there is no real solution with C language for
that.
So, the only thing I can see is GCC related, or a parsing script using
sed/awk for instance...
If anyone has some tips I am interested .

JM

 

[James Kuyper]

On 11/23/2011 3:41 PM, James Kuyper wrote: ....
Are you sure you are not mixing up sizeof type and sizeof object?

Yes, I'm sure. Would you care to explain why you think I might be?

"sizeof(type)" is standard feature of C. "sizeof expression" is another
standard feature of C, essentially equivalent
"sizeof(typeof(expression))", except that standard C doesn't have "typeof".
"sizeof object" is not a feature of C, though "sizeof
expression_referring_to_object" can be misinterpreted as having that
meaning.

 

[James Kuyper]

Am 23.11.2011 21:41, schrieb James Kuyper: ....

The C standard does mandate the sizeof operator, and it can be used to
create a program which, at run-time, will generate output like that
which he's asking for; I said as much in a later part of my message
which you snipped.

The C standard does not mandate the existence of any feature that would
produce such output at compile time. The only output that normally
appears at compile time and is mandated by the C standard is diagnostic
messages, the text of which is not specified by the standard, nor under
the control of the developer - with the exception of the #error
directive. However, the #error directive is executed during phase 4;
sizeof expressions cannot be evaluated until phase 7, so the value of
such expressions cannot appear in #error output.

Well, sizeof, unless applied to a variable-length array, yields an
integer constant. Which AFAIK, can be used e.g. as array bound. That's
how I understood the OP's "would like, at compile time ideally, […], to
know the footprint in memory" requirement.

I agree that, when VLAs are not involved, the compiler has sufficient
information to produce the result that he wants; I'm just letting him
know the C standard doesn't mandate support for any feature that would
have the desired effect.

 

[Jean-Michel Hautbois]

Am 23.11.2011 21:41, schrieb James Kuyper: ...

Well, sizeof, unless applied to a variable-length array, yields an
integer constant. Which AFAIK, can be used e.g. as array bound. That's
how I understood the OP's "would like, at compile time ideally, […], to
know the footprint in memory" requirement.

I agree that, when VLAs are not involved, the compiler has sufficient
information to produce the result that he wants; I'm just letting him
know the C standard doesn't mandate support for any feature that would
have the desired effect.

Taking both of your remarks into account, I am understanding that
#error with sizeof can produce the output I want at compile time.
Then, my only problem is to say : how can I "automate" (didn't find a
better word) the structure parsing/sizeof ?
In short, on the example I gave in my OP, how would you generate an
output like the one I gave ?
I can make a macro, which would display sizeof() result using #error,
then call this macro for each element of my structure.
But each time I want to add a new element, I have to add an explicit
call to that macro.
And I can't figure out a way to do it...

JM

 

[Ben Bacarisse]

Am 23.11.2011 21:41, schrieb James Kuyper: ...
The C standard does mandate the sizeof operator, and it can be used to
create a program which, at run-time, will generate output like that
which he's asking for; I said as much in a later part of my message
which you snipped.

The C standard does not mandate the existence of any feature that would
produce such output at compile time. The only output that normally
appears at compile time and is mandated by the C standard is diagnostic
messages, the text of which is not specified by the standard, nor under
the control of the developer - with the exception of the #error
directive. However, the #error directive is executed during phase 4;
sizeof expressions cannot be evaluated until phase 7, so the value of
such expressions cannot appear in #error output.

Well, sizeof, unless applied to a variable-length array, yields an
integer constant. Which AFAIK, can be used e.g. as array bound. That's
how I understood the OP's "would like, at compile time ideally, […], to
know the footprint in memory" requirement.

I agree that, when VLAs are not involved, the compiler has sufficient
information to produce the result that he wants; I'm just letting him
know the C standard doesn't mandate support for any feature that would
have the desired effect.

It's betst tosnip sigs.

Taking both of your remarks into account, I am understanding that
#error with sizeof can produce the output I want at compile time.

I don't think anyone has said this. James says the opposite in the text
you quote.

Then, my only problem is to say : how can I "automate" (didn't find a
better word) the structure parsing/sizeof ?
In short, on the example I gave in my OP, how would you generate an
output like the one I gave ?

The only reasonable solution has already been suggested: write a program
that can pick out structure definitions, parse them, and generate
another C program that prints the desired output.

This is enough work that the output would have to be of significant
value. Is that th case? Why do you need this automated output?

I can make a macro, which would display sizeof() result using #error,
then call this macro for each element of my structure.
But each time I want to add a new element, I have to add an explicit
call to that macro.
And I can't figure out a way to do it...

You can't do it with the C pre-processor.

 

[BartC]

The only reasonable solution has already been suggested: write a program
that can pick out structure definitions, parse them, and generate
another C program that prints the desired output.

This is enough work that the output would have to be of significant
value. Is that th case? Why do you need this automated output?

If the overall structure is simple: one top-level struct, multiple
second-level structs only containing char arrays, and with a guaranteed
layout, then a utility is not that difficult to write. Perhaps, within a
much larger body of code, it should be surrounded by guards:

/* --- Start --- */
....
/* --- End --- */

which can be easily scanned for by some program. However I don't recommend
using C for such a task. I've just had a go myself, and on the code fragment
the OP posted, it output this table:

t_buffer_s ( 196608 bytes)
t_buffer_head ( 131072 bytes)
buffer1 ( 65536)
buffer2 ( 65536)
t_buffer_subhead ( 65536 bytes)
buffer1 ( 32768)
buffer2 ( 32768)

But it makes a *lot* of assumptions. If the structure is going to be much
more variable, then a proper C parser is needed, or the input has to follow
a very stylised format, or perhaps mini-comments can be added on each line
to guide the utility.

 

[James Kuyper]

Note the use of the word "not".

Note the use of the word "cannot".

Well, sizeof, unless applied to a variable-length array, yields an
integer constant. Which AFAIK, can be used e.g. as array bound. That's
how I understood the OP's "would like, at compile time ideally, [�], to
know the footprint in memory" requirement.

I agree that, when VLAs are not involved, the compiler has sufficient
information to produce the result that he wants; I'm just letting him
know the C standard doesn't mandate support for any feature that would

Note the use of the word "doesn't".

....
Taking both of your remarks into account, I am understanding that
#error with sizeof can produce the output I want at compile time.

I curious how you reached that conclusion, because what I said was
almost precisely the opposite. Well, not exactly - the standard doesn't
mandate support for any features that could be counted on to produce the
result that you want at compile time, but neither does it prohibit them.

 

[Joe keane]

"nm" gives you some help although not what OP asked for.

"gprof" gives you some help although not what OP asked for.

You can try something like:

#define malloc(SZ) ((SZ) < 512 ? malloc_some_small_thing(SZ) :
(SZ) < 8192 ? malloc_some_medium_thing(SZ) :
malloc_some_big_thing(SZ))